Problem Note 64346: Using PROC APPEND to append millions of rows with values that fail the integrity constraints causes the SAS job to seem to stop responding
 |  |  |  |
The APPEND procedure adds all row data to the BASE= data set before integrity constraint validation is performed and before data values are added to indexes. This approach of appending data is called the fast-append method, and it is the default method for PROC APPEND.
If a row’s data values violate one of the defined integrity constraints, the row must be removed from the BASE= data set. When a large number of rows fails the integrity constraint validation step, the time required for removing each row from the BASE= data set uses significant computer resources. Removing large numbers of rows can cause the SAS job to seem to stop responding.
In most cases, using PROC APPEND to append millions of invalid rows of data to a BASE= data set should still complete successfully with a warning in the SAS® log. A message similar to the following indicates how many invalid rows of data were not added to the BASE= data set.
NOTE: There were 9453683 observations read from the data set YYY.YYYY.
NOTE: 0 observations added.
NOTE: The data set XXX.XXXX has 9578091 observations and 11 variables.
NOTE: PROCEDURE APPEND used (Total process time):
real time 1:15.63
cpu time 43.70 seconds
However, this warning does not appear if the THREADS system option is in effect. When multiple threads are used during the PROC APPEND process, the SAS job seems to stop responding, rather than to end with a warning or error.
Two approaches can address this PROC APPEND issue.
Solution 1: Use the APPENDVER=V6 Option
The APPENDVER=V6 option appends one observation at a time to the BASE= data set. The integrity constraint validation step is performed on each row before the row is added to the data set. This option prevents the rows from having to be removed from the data set if they fail the integrity constraint validation step. Using the APPENDVER=V6 option prevents the default fast-append method from being used.
Solution 2: Use the NOTHREADS System Option
To work around issues that occur when multiple threads are used during PROC APPEND processing, use the NOTHREADS system option to turn threading off.
Operating System and Release Information
| Product Family | Product | System | Product Release | SAS Release | ||
| Reported | Fixed* | Reported | Fixed* | |||
| SAS System | Base SAS | z/OS | 9.3_M2 | 9.3 TS1M2 | ||
| z/OS 64-bit | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft® Windows® for x64 | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows 8 Enterprise 32-bit | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows 8 Enterprise x64 | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows 8 Pro 32-bit | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows 8 Pro x64 | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows 8.1 Enterprise 32-bit | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows 8.1 Enterprise x64 | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows 8.1 Pro 32-bit | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows 8.1 Pro x64 | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows Server 2003 Datacenter Edition | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows Server 2003 Enterprise Edition | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows Server 2003 Standard Edition | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows Server 2003 for x64 | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows Server 2008 | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows Server 2008 R2 | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows Server 2008 for x64 | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows Server 2012 Datacenter | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows Server 2012 R2 Datacenter | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows Server 2012 R2 Std | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows Server 2012 Std | 9.3_M2 | 9.3 TS1M2 | ||||
| Microsoft Windows XP Professional | 9.3_M2 | 9.3 TS1M2 | ||||
| Windows 7 Enterprise 32 bit | 9.3_M2 | 9.3 TS1M2 | ||||
| Windows 7 Enterprise x64 | 9.3_M2 | 9.3 TS1M2 | ||||
| Windows 7 Home Premium 32 bit | 9.3_M2 | 9.3 TS1M2 | ||||
| Windows 7 Home Premium x64 | 9.3_M2 | 9.3 TS1M2 | ||||
| Windows 7 Professional 32 bit | 9.3_M2 | 9.3 TS1M2 | ||||
| Windows 7 Professional x64 | 9.3_M2 | 9.3 TS1M2 | ||||
| Windows 7 Ultimate 32 bit | 9.3_M2 | 9.3 TS1M2 | ||||
| Windows 7 Ultimate x64 | 9.3_M2 | 9.3 TS1M2 | ||||
| Windows Vista | 9.3_M2 | 9.3 TS1M2 | ||||
| Windows Vista for x64 | 9.3_M2 | 9.3 TS1M2 | ||||
| 64-bit Enabled AIX | 9.3_M2 | 9.3 TS1M2 | ||||
| 64-bit Enabled HP-UX | 9.3_M2 | 9.3 TS1M2 | ||||
| 64-bit Enabled Solaris | 9.3_M2 | 9.3 TS1M2 | ||||
| HP-UX IPF | 9.3_M2 | 9.3 TS1M2 | ||||
| Linux | 9.3_M2 | 9.3 TS1M2 | ||||
| Linux for x64 | 9.3_M2 | 9.3 TS1M2 | ||||
| Solaris for x64 | 9.3_M2 | 9.3 TS1M2 | ||||
| Type: | Problem Note |
| Priority: | medium |
| Date Modified: | 2019-06-21 12:56:25 |
| Date Created: | 2019-06-17 18:46:14 |