Data Compression

SAS LASR Analytic Server supports compression for in-memory tables. All the analytic statements, such as PERCENTILES, LOGISTIC, and so on, in the IMSTAT procedure are supported for compressed tables as well as regular, uncompressed, tables. Clients like SAS Visual Analytics can also operate on compressed tables as well.

All compression is performed by the server. In other words, when you transfer a table to the server in a DATA step and specify the SQUEEZE= data set option, the rows are sent to the server as is, and the server compresses the rows. The server uses the zlib compression algorithm that is described in RFC 1950, "ZLIB Compressed Data Format Specification."

All data in a row, both character and number variables, are compressed. Every row in a table is compressed, the server does not support some rows in compressed form and others as uncompressed. The server can report the uncompressed size of the table, the compressed size, and the compression ratio.

For matrices of computed doubles (with lots of decimal places), compression might not reduce the storage requirements at all. For rows with many long character variables that consist mostly of blanks, the compression ratio can be very high. For rows with mixed variables where most doubles do not have fractional parts and most character variables have a small amount of blank padding, the compression ratio is typically moderate. As with most cases of using compression, character variables tend to compress the most and the ratio depends on your data.

The following example shows how to use the SQUEEZE= data set option for the SAS LASR Analytic Server.

libname example sasiola host="grid001.example.com" port=10010 tag=hps;

data example.prdsale(squeeze=yes);
    set sashelp.prdsale;
run;

After the table is loaded to memory, you can access the compressed table with the Example.Prdsale libref.

The server supports the APPEND= option for compressed tables. The following example shows how to add new rows (uncompressed) to the compressed table:

data example.prdsale(append=yes);
    somelib.newrows;
run;

Because the Example.Prdsale table is already compressed, the new rows are automatically compressed as they are appended to the table. Specifying SQUEEZE= with APPEND= has no effect. If the table is compressed, the server compresses the new rows. If the table is not compressed, the server does not compress the new rows (even if SQUEEZE=YES is specified). The compressed or uncompressed state of the table determines how the rows are appended.

Partitioning and compression are supported together. The following example creates a new in-memory table that is partitioned and compressed:

data example.iris(partition=(species) squeeze=yes);
    set sashelp.iris;
run;

data example.iris(append=yes);
    set somelib.moreirises;
run;

In the first DATA statement, the Iris data set is loaded to memory on the server and is partitioned by the formatted values of the Species variable. The table is also compressed. In the second DATA statement, the table is appended to with more rows. Because the in-memory table is already partitioned and compressed, the new rows are automatically partitioned and compressed when they are appended.

The LASR procedure is used for loading data to memory on distributed SAS LASR Analytic Server.

The following example shows how to read a SAS data set and compress it in memory on the server:

proc lasr add data=sashelp.prdsale port=10010 squeeze;
    performance host="grid001.example.com";
run;

The example uses the SQUEEZE option to read the Prdsale data set from the Sashelp library and compress it in-memory on the server. Be aware that you must specify the SQUEEZE option for each table that you want to load in compressed form. You cannot specify SQUEEZE with the CREATE option when you start a server and have the server automatically compress all tables.

The SQUEEZE option works when reading data from SAS/ACCESS engines, too. The resulting in-memory table is compressed whether you read data serially from a standard database or you read data in parallel from a distributed database like Greenplum Database.

When you read SASHDAT tables into memory, the compression for the resulting in-memory tables depends on the following:

If you specify a WHERE clause and the SQUEEZE option, then the server evaluates the WHERE clause as it reads data from HDFS and compresses the rows that meet the WHERE clause criteria. The memory efficiencies of the SASHDAT table format are forfeited in this scenario because the server had to apply the WHERE clause.

If you do not specify a WHERE clause, then the server ignores the SQUEEZE option and relies on whether the SASHDAT table is compressed. If the SASHDAT table is compressed, then the in-memory representation of the table is also compressed. If the SASHDAT table is not compressed, then the in-memory representation is not compressed either. The server ignores the option so that it can keep the memory efficiencies of the SASHDAT table format—when a SASHDAT table is loaded to memory, the in-memory representation is identical to the on-disk representation.

Compression exchanges less memory use for more CPU use. It slows down any request that processes the data. An in-memory table consists of blocks of rows. When the server works with a compressed table, the blocks of rows must be uncompressed before the server can work with the variables. In some cases, a request can require five times longer to run with a compressed table rather than an uncompressed table.

For example, if you want to summarize two variables in a table that has 100 variables, all 100 columns must be uncompressed in order to locate the data for the two variables of interest. If you specify a WHERE clause, then the server must uncompress the data before the WHERE clause can be applied. Like the example where only two of 100 variables are used, if the WHERE clause is very restrictive, then there is a substantial performance penalty to filter out most of the rows.

Working with SASHDAT tables that are loaded from HDFS is the most memory-efficient way to use the server. Using compressed SASHDAT tables preserves the memory efficiencies, but still incurs the performance penalty of uncompressing the rows as the server operates on each row..

The limitations for working with compressed tables are as follows: