Converting SPD Server 3.x Tables to SPD Server 4.x Tables

Before You Convert

The SPDSCONV utility changes the format of SPD Server 3.x tables to the format used by SPD Server 4.x tables. The conversion is not reversible. After you convert a table for use with SPD Server 4.x, the table can no longer be read by SPD Server 3.x. Before you start converting SPD Server tables, back up all your existing SPD Server 3.x tables. If you ever need to revert to your SPD Server 3.x tables, you can restore them.
If you need to temporarily use your tables with both SPD Server 3.x and SPD Server 4.x, you can use PROC COPY to make copies of the tables before you convert them. You can use SPDSCONV on the copied versions to convert them for use with SPD Server 4.x (and, by extension, SPD Server 5.1) and maintain archival versions of the older SPD Server tables.

Overview of the SPDSCONV Utility

The SPDSCONV utility converts SPD Server 3.x metadata files for use with SPD Server 4.x. The conversion process updates the physical structure of the metadata files and renames them. The SPDSCONV utility also updates the data partition files if the SPD Server 3.x tables that are being converted contain compressed data.
You can identify SPD Server 3.x table files by their filename extension. SPD Server 3.x table files end with the filename extension .spds8. SPD Server 4.x and SPD Server 5.1 table files end with the filename extension .spds9. All tables that are upgraded to be compatible with SPD Server 4.x or SPD Server 5.1 have the filename extension .spds9.
SPD Server 4.x and SPD Server 5.1 index files differ from SPD Server 3.x index files. SPD Server 4.x and SPD Server 5.1 index files permit greater numbers of observations than SPD Server 3.x index files did. SPD Server 3.x index files are not compatible with the SPD Server 4.x or SPD Server 5.1 environment.
The SPDSCONV table conversion utility does not re-create index files. When you use the SPDSCONV utility to convert tables from SPD Server 3.x to SPD Server 4.x and SPD Server 5.1 format, the utility automatically deletes physical files that were associated with the old 3.x indexes and that are now obsolete. The SPDSCONV utility does offer an option to create a SAS job file that you can run in the SPD Server 4.x or SPD Server 5.1 environment to re-create the SPD Server 3.x index files for use with SPD Server 5.1.
If you choose to create the SAS job file to re-create SPD Server 3.x indexes for use in SPD Server 4.x and SPD Server 5.1, the code resembles the following example:
%let SPDSIASY=YES;
 PROC DATASETS 
  lib=<spdsv4 LIBNAME>;
 modify MYTABLE;
   index create X1 [/Options];
   index create X2 [/Options];
 ...

 quit;
You can specify the destination directory for the SAS job file that you create, but the SPDSCONV utility names the job file that you create. The utility generates SAS job filenames by adding the text string _v4ix.sas to the table name.
Tip
You should defer re-creating an index because the process can require intensive computing. In busy computing environments, consider performing this task as an off-peak batch job.
After you convert the indexes, you might notice that SPD Server 4.x metadata files are slightly larger than the SPD Server 3.x metadata files were. The increase in file size is related to the new structures that enable SPD Server 4.x and SPD Server 5.1 to use tables that contain more than 2,000,000 rows of data.
How does SPDSCONV work? When the SPDSCONV utility converts a table, it reads the original SPD Server 3.x metadata file and creates a new SPD Server 4.x and SPD Server 5.1 metadata file. Both these files are locked during the conversion process. The lock prevents other users from accessing the files while changes are being made. If the conversion process encounters problems, the SPD Server 4.x and SPD Server 5.1 metadata file is deleted, and the original SPD Server 3.x table remains intact.
SPDSCONV reads the SPD Server 3.x metadata file one section at a time, and re-creates each structure in the SPD Server 4.x and SPD Server 5.1 metadata file as it is read. After the SPD Server 4.x and SPD Server 5.1 metadata file is fully populated, the utility checks the data partition file component to determine whether updates are required.
If SPDSCONV detects the presence of compression block headers, then the data partition file contains SAS 8 compression information that is not compatible in SPD Server 4.x and SPD Server 5.1, and SPDSCONV must update the data partition files. SPDSCONV updates the file by overwriting the compression block headers. SPDSCONV does not change the size of the data partition file, of any file components, or of any data that is contained in the files. The increase in metadata file size is modest and represents only a small percentage of storage space when compared to its corresponding data partition file component.
After SPDSCONV updates the data partition file, you cannot restore or re-create the original SPD Server 3.x data partition file. Ensure that you have complete backup images of the SPD Server 3.x data sets that you intend to convert before you run the conversion.
After the SAS job file re-creates the SPD Server 3.x indexes for use with SPD Server 4.x and SPD Server 5.1, all remnants of the SPD Server 3.x table are deleted. The SPDSCONV utility does not perform ACL checks during the conversion process. You cannot browse the contents of table rows from within the utility. During the metadata file conversion, no table rows are accessed, and there are no options to expose table row contents as part of logging or index job creation. The SPD Server 4.x and SPD Server 5.1 table retains the same SPD Server owner as the SPD Server 3.x table.

Using SPDSCONV

The SPDSCONV program is a command-line utility. You use a set of command-line options and parameters to specify the name and location of tables that you want to convert. Then you specify the options that you want for your conversion. If your SPD Server software is installed on a UNIX platform, see Installing SAS Scalable Performance Data (SPD) Server on UNIX for information about setting up your environment before you run SPDSCONV.
The command-line syntax is as follows:
SPDSCONV <Options> [-a | table1 [table2]] 
The order of options and table names on the command line does not matter. All of the currently available options are global options. Placing a global option before or after a table does not change the option setting for that table alone. Global options are always applied globally.
The SPDSCONV command has the following options:
-d pathname
the directory path that corresponds to an existing SPD Server LIBNAME domain. This pathname should be the same as the PATHNAME= directory path in the libnames.parm file.
-l logpath
the directory path where SAS job files that are created during the conversion process should be stored. The default logpath setting is the directory from which the SPDSCONV command is issued.
-a
converts all SPD Server 3.x compatible tables in the -d pathname directory to tables that are compatible with SPD Server 4.x and SPD Server 5.1.
-j
creates a SAS job in the log directory for each SPD Server 3.x table that contains an index. When you use the -j option, the SAS job re-creates the indexes on the SPD Server 5.1 table. You must run the SAS job after the SPDSCONV utility completes the conversion process. Because re-creating an index recreation can be computation-intensive, you should schedule SAS index re-creation jobs as a SAS batch job during off-peak hours. The utility generates the name of the SAS job file, which has the following format:
TableName_v4ix.sas
In the name, TableName is the name of the table that is compatible with SPD Server4.x and SPD Server 5.1. The SAS job file contains the SAS language statements that are required for re-creating the indexes that the SPD Server 3.x table used. You need to edit the job file before the job executes. Editing the file ensures that the correct SPD Server 4.x and SPD Server 5.1 LIBNAME is used with the PROC DATASETS statement.
-v
creates verbose output for the conversion process.

SPDSCONV Utility Examples

Example LIBNAME Parameter File

These examples assume that you use the following LIBNAME parameter file for your SPD Server installation:
LIBNAME=usmkt pathname=/mdat1/usmkt
 roptions="datapath=('/dat11/usmkt'
                    '/dat12/usmkt'
                    '/dat13/usmkt'/
                    '/dat14/usmkt')
 indexpath=('/ix11/usmkt'
            '/ix12/usmkt')";

LIBNAME=sales pathname=/mdat1/sales
 roptions="datapath=('/dat21/sales'
                    '/dat22/sales'
                    '/dat23/sales'/
                    '/dat24/sales')

 indexpath=('/ix21/sales'
            '/ix22/sales')"; 

Converting a Simple Table

Suppose you have an SPD Server 3.x table named CT010299 that belongs to the usmkt domain, and you want to convert CT010299 to use with SPD Server 4.x and SPD Server 5.1. Table CT010299 does not have indexes and you want a verbose output of the table conversion.
Issue the following command:
SPDSCONV -v -d /mdat1/usmkt CT010299

Converting Tables and Re-creating Indexes

Suppose you want to convert all tables in the sales domain. You also want SPDSCONV to create SAS jobs that you can run to re-create the indexes after the table conversion is complete. You want the SAS jobs put into the directory $HOME/salesv9. You also want a verbose output of the conversion.
Issue the following command:
SPDSCONV -v -d /mdat1/sales -l $HOME/salesv9 -j -a