Overview: The SPD Engine

Comparing the Default Base SAS Engine and the SPD Engine

Default Base SAS engine data sets and SPD Engine data sets have many similarities. They both store data in a SAS library, which is a collection of files that reside in one or more directories. However, because the SPD Engine data libraries can span devices and file systems, the SPD Engine is ideal for use with very large data sets. Also, the SPD Engine enables you to specify separate directories, or devices, for each component in the LIBNAME statement. Creating and Loading SPD Engine Files provides details on designing and setting up the SPD Engine data libraries.

The SPD Engine Libraries and File Systems

An SPD Engine library can contain data files, metadata files, and index files. The SPD Engine does not support catalogs, SAS views, MDDBs, or other utility (byte) files.

The SPD Engine uses the zFS file system for z/OS and the ODS-5 file system for OpenVMS on HP Integrity Servers. This means that some functionality might be slightly different on these platforms. For example, for z/OS, the user must have a home directory on zFS.

Utility File Workspace

Utility files are generated during the SPD Engine operations that need extra space (for example, when creating parallel indexes or when sorting very large files). Default locations exist for all platforms but, if you have large amounts of data to process, the default location might not be large enough. The SPD Engine system option SPDEUTILLOC= lets you specify a set of file locations in which to store utility scratch files. For more information, see SPDEUTILLOC= System Option.

Temporary Storage of Interim Data Sets

To create a library to store interim data sets, specify the SPD Engine option TEMP= in the LIBNAME statement. If you want current applications to refer to these interim files using one-level names, specify the library on the USER= system option.

The following example code creates a user libref for interim data sets. It is deleted at the end of the session.

libname user spde '/mydata' temp=yes; 
data a; x=1; 
run; 
proc print data=a;

The USER= option can be set in the configuration file so that applications that reference interim data sets with one-level names can run in the SPD Engine.

Differences between the Default Base SAS Engine Data Sets and the SPD Engine Data Sets

The following chart compares the SPD Engine capabilities to default Base SAS engine capabilities.

Comparing the Default Base SAS Engine Data Sets and the SPD Engine Data Sets
Feature	SPD Engine	Default Base SAS Engine
Partitioned data sets	yes	no
Parallel WHERE optimization	yes	no
Lowest locking level	member	record
Concurrent access from multiple SAS sessions on a given data set	READ (INPUT open mode)	READ and WRITE (all open modes)
Remote computing via SAS/CONNECT	no	yes
Data transfer via SAS/CONNECT	no	yes
RLS (Remote Library Services) via SAS/CONNECT	no	yes
Available via SAS/CONNECT	no	yes
Support in SAS/SHARE	no	yes
Automatic sort for SAS BY processing (sort a temporary copy of the data to support BY processing)	yes	no
User-defined formats and informats	yes, except in WHERE (table note 1)	yes
Catalogs	no	yes
Views	no	yes
MDDBs	no	yes
Integrity constraints	no	yes
Data set generations	no	yes
CEDA	no	yes
Audit trail	no	yes
NLS transcoding	no	yes
Number of observations that can be counted	2⁶³-1 (on all hosts)	2³¹-1 (on 32-bit hosts)2⁶³-1 (on 64-bit hosts)
COMPRESS=	YES\|NO\|CHAR\|BINARY (only if the file is not encrypted)	YES\|NO\|CHAR\|BINARY
ENCRYPT=	cannot be used with COMPRESS=	can be used with COMPRESS=
Encryption	data files only	yes (all files)
FIRSTOBS= system option and data set option	no	yes
OBS= system option and data set option	yes, if used without ENDOBS= or STARTOBS= SPD Engine options	yes
Functions and call routines	yes, with some exceptions (table note 2)	yes
Move table via OS utilities to a different directory or folder	no	yes
Observations returned in physical order	no, if BY or WHERE is present	yes

TABLE NOTE 1: In WHERE processing, user-defined formats and informats are passed to the supervisor for handling. Therefore, they are not processed in parallel.

TABLE NOTE 2: In WHERE processing, functions and call routines introduced in SAS 9 or later are passed to the supervisor for handling. Therefore, they are not processed in parallel.

Top of Page