Extending the Observation Count in a SAS Data File

Definition of Extended Observation Count

Extending the observation count in a SAS data file is a process that creates an enhanced file format that counts observations beyond the 32-bit long integer maximum. By specifying the EXTENDOBSCOUNTER= option when you create an output SAS data file, the resulting 32-bit file behaves like a 64-bit file regarding counters. A SAS data file with an extended observation count, however, is incompatible with releases prior to SAS 9.3, and the extended observation count file attribute cannot be inherited.

To understand the maximum observation count, see Understanding the Observation Count in a SAS Data File.

Requesting to Extend the Observation Count

Use the EXTENDOBSCOUNTER= Option

To extend the observation count in an output SAS data file, use the EXTENDOBSCOUNTER= option. The use of this option requests an enhanced file format that counts observations beyond the 32-bit long maximum. By specifying the option when the file is created, the resulting 32-bit file behaves like a 64-bit file regarding counters.

By default, the EXTENDOBSCOUNTER= functionality is set to NO. This setting specifies that the maximum observation count is determined by the long integer size for the operating environment. To extend the observation count, specify EXTENDOBSCOUNTER=YES as either a SAS data set option or a LIBNAME statement option.

As a SAS data set option, EXTENDOBSCOUNTER=YES causes the newly created output file to be created with an extended observation count.
As a LIBNAME statement option, EXTENDOBSCOUNTER=YES causes newly created output files in the SAS library to be created with an extended observation count.

For example, when processed in an operating environment with a 32-bit long integer, the following code creates an output SAS data file that extends the observation count beyond the 32-bit long maximum. The EXTENDOBSCOUNTER= option is specified as a data set option in the DATA statement. This change requests the enhanced file format for the individual file:

libname myfiles 'C:\MyFiles';

data myfiles.bigfile (extendobscounter=yes);
   .
   .
   .
run;

Specifying EXTENDOBSCOUNTER= as a LIBNAME statement option requests the enhanced file format for all files created for the SAS library. The following code uses the COPY procedure to recreate all of the files in the SAS library with the extended observation count:

libname new 'C:\NewFiles' extendobscounter=yes;
libname old 'C:\OldFiles'; 

proc copy in=old out=new;
run;

Tip

Specifying the EXTENDOBSCOUNTER= option in a LIBNAME statement is also useful to migrate a library with the MIGRATE procedure, which does not support SAS data set options.

For details about the EXTENDOBSCOUNTER= option syntax, see the EXTENDOBSCOUNTER= Data Set Option in SAS Data Set Options: Reference and EXTENDOBSCOUNTER=NO | YES in SAS Statements: Reference.

Recovering from an Exceeded Maximum Observation Count

To recover a SAS data file that exceeds the maximum number of observations that can be counted, you can recreate the SAS data file with an extended observation count by specifying the EXTENDOBSCOUNTER= option. One method to convert a file that exceeds the maximum observation count is to use the DATA step SET statement. For example, the following code creates a new file with an extended observation count by making a copy of the existing file:

libname lib 'C:\Myfiles';

data lib.b (extendobscounter=yes);
   set lib.a;
run;

Extended Observation Count File Attribute

When a SAS data file is created with EXTENDOBSCOUNTER=YES, the file contains an attribute to indicate that the file format is enhanced to extend the observation count. For example, the following CONTENTS procedure output lists ExtendObsCounter information under Engine/Host Dependent Information.

Note: If a SAS data file does not contain the extended observation count file attribute, the ExtendObsCounter field is not listed.

CONTENTS Procedure Output Showing ExtendObsCounter Attribute

Extended Observation Count Behavior Considerations

A SAS data file that contains the extended observation count attribute is supported only in SAS 9.3. The enhanced file format is incompatible with releases prior to SAS 9.3. If you attempt to open a SAS data file that contains the extended observation count attribute in a SAS release before SAS 9.3, an error message occurs. For example:

ERROR: File MYFILES.EXTEND.DATA not compatible with this SAS version.

In addition, the extended observation count attribute cannot be inherited by a new file. To create a file with an extended observation count from a file that contains the extended observation count attribute, specify the EXTENDOBSCOUNTER= option for the new file. For example, the following code creates a new file named EXTEND2 from an existing file named EXTEND1, which contains the extended observation count attribute. The DATA statement for the new file must specify the EXTENDOBSCOUNTER= data set option:

data extend2 (eoc=yes);
   set extend1;
run;

Operations that are affected by the extended observation count attribute include the following:

SAS functionality that copies files (such as the APPEND procedure, COPY procedure, MIGRATE procedure, and SET statement) does not copy the extended observation count attribute.
If you assign the USER library, which enables SAS files with one-level names to be stored in a permanent SAS library, the method that you use to associate the USER libref with a SAS library determines whether output files that are created for the library contain the extended observation count attribute. If you directly assign the USER libref with the LIBNAME statement or the LIBNAME function and you specify the EXTENDOBSCOUNTER= option, files that are created for the USER library contain the extended observation count attribute.
However, if you first assign a libref to a SAS library with the LIBNAME statement, and then use the USER= system option to specify that SAS library as the default for one-level names, files that are created for the USER library do not contain the extended observation count attribute. Even though the LIBNAME statement can specify the EXTENDOBSCOUNTER= option, the USER libref does not inherit the extended observation count attribute from the LIBNAME statement libref. For example, the file that is created with the following code does not contain the extended observation count attribute.
```
libname testdata 'C:\Myfiles' extendobscounter=yes;
options user=testdata;

data user.newfile;
   .
   .
   .
run;
```
In a SAS/SHARE client session, the EXTENDOBSCOUNTER= option in the LIBNAME statement is ignored if it is specified in combination with the SERVER= option. To create output SAS data files with the extended observation count attribute, use the EXTENDOBSCOUNTER= data set option.
When you specify the FIRSTOBS= or OBS= option for a SAS data file that is created with EXTENDOBSCOUNTER=YES, the performance is improved for a file with 2³¹-1 observations or more in a 32-bit environment.

Using the EXTENDOBSCOUNTER= Option in a 64-Bit Operating Environment

In an operating environment that stores the observation count as a 64-bit long integer, you do not need to specify the EXTENDOBSCOUNTER= option. For example, when processed in an operating environment with a 64-bit long integer such as HP-UX on 64-bit Itanium, the following code results in a message to the SAS log:

libname myfiles '/u/myid/myfiles';

data myfiles.bigfile (extendobscounter=yes);
   .
   .
   .
run;

NOTE: EXTENDOBSCOUNTER=YES is ignored on a SAS data set with a 64-bit long 
observation counter.

However, if you are using the OUTREP= option to create an output file with a 32-bit data representation, the observation count is stored as a 32-bit long. In this case, you might want to specify EXTENDOBSCOUNTER=YES in order to create a 32-bit file that behaves like a 64-bit file with respect to counters. For example, the following code is submitted in the 64-bit operating environment HP-UX on 64-bit Itanium and requests a Microsoft Windows 64-Bit Edition data representation, which uses the 32-bit model long integer data type.

libname myfiles '/u/MyFiles';

data myfiles.bigfile (outrep=windows_64 extendobscounter=yes);
   .
   .
   .
run;

Tip

Even though Microsoft Windows 64-Bit Edition is a 64-bit operating environment, the observation count is stored as a 32-bit long integer. Therefore, you might want to extend the observation count for SAS data files in the Microsoft Windows 64-Bit Edition operating environment.

When to Use the EXTENDOBSCOUNTER=YES Option

Specify EXTENDOBSCOUNTER=YES only for an output SAS data file whose internal data representation stores the observation count as a 32-bit long integer.

In an operating environment that stores the observation count as a 64-bit long integer, you do not need to specify EXTENDOBSCOUNTER=YES unless you are creating an output file with a 32-bit data representation by specifying the OUTREP= option.
In an operating environment that stores the observation count as a 32-bit long integer, if you are creating an output file with a 64-bit representation by specifying the OUTREP= option, you do not need to specify EXTENDOBSCOUNTER=YES.

The following table lists the SAS 9.3 operating environments, the data representation value for each operating environment, the default observation count size, and whether specifying EXTENDOBSCOUNTER=YES is appropriate. For example, if your operating environment is AIX on 64-bit platform, you do not need to specify EOC=YES. However, in the AIX on 64-bit platform, if you use the OUTREP= option to specify the data representation value LINUX_32, then you can specify EXTENDOBSCOUNTER=YES.

When to Use EXTENDOBSCOUNTER=YES

SAS 9.3 Operating Environment	Data Representation Value	Default Observation Count Size	EOC=YES
AIX on 64-bit platform	RS_6000_AIX_64	64-bit counter	no
HP-UX for PA-RISC on 64-bit platform	HP_UX_64	64-bit counter	no
HP-UX for Itanium on 64-bit platform	HP_IA64	64-bit counter	no
Linux for Intel architecture	LINUX_32	32-bit counter	yes
Linux for x64	LINUX_X86_64	64-bit counter	no
Microsoft Windows on 32-bit platform	WINDOWS_32	32-bit counter	yes
Microsoft Windows 64-Bit Edition	WINDOWS_64	32-bit counter	yes
Solaris for x64	SOLARIS_X86_64	64-bit counter	no
Solaris on SPARC 64-bit platform	SOLARIS_64	64-bit counter	no
32-bit SAS on z/OS	MVS_32	32-bit counter	yes