Problem Note 61812: In the CASUTIL procedure, when you run parallel LOAD statements from Hadoop, the load fails with various error messages
 |  |  |  |  |
In SAS® Viya® 3.3, a parallel LOAD statement from Hadoop in PROC CASUTIL might fail if the source table contains multiple consecutive records with missing variable values. The error shown in the SAS log might differ depending on the amount of data processed.
Here is an example SAS log:
caslib hive01 datasource=(srctype="hadoop" server="server.example.com" schema="default"
dataTransferMode="parallel"
hadoopconfigdir="/opt/sas/viya/config/data/hadoop/conf"
hadoopjarpath="/opt/sas/viya/config/data/hadoop/lib");
NOTE: 'HIVE01' is now the active caslib.
NOTE: Cloud Analytic Services added the caslib 'HIVE01'.
NOTE: Action to ADD caslib HIVE01 completed for session MYSESSION.
/* 100000 rows */
proc casutil;
NOTE: The UUID 'fd16bf47-3bd3-4e44-8eef-85dce5bb9537' is connected using session MYSESSION.
load casdata= "100krows" casout= "100krows"
incaslib="hive01" outcaslib="hive01" copies=0 replace;
NOTE: Performing parallel LoadTable action using SAS Data Connect Accelerator for Hadoop.
ERROR: The Embedded Process job has failed. Please see job log for details. Job tracking URL :
http://server.example.com:8088/proxy/application_1515449747420_0026/.
NOTE: Action failed during Embedded Process processing. Not retrying action due to dataTransferMode setting of parallel.
ERROR: An error occurred executing SAS Embedded Process job.
ERROR: The action stopped due to errors.
NOTE: The Cloud Analytic Services server processed the request in 39.89334 seconds.
quit;
NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE CASUTIL used (Total process time):
real time 39.90 seconds
cpu time 0.00 seconds
/* 1000 rows */
proc casutil;
NOTE: The UUID 'fd16bf47-3bd3-4e44-8eef-85dce5bb9537' is connected using session MYSESSION.
load casdata= "1krows" casout= "1krows"
incaslib="hive01" outcaslib="hive01" copies=0 replace;
NOTE: Performing parallel LoadTable action using SAS Data Connect Accelerator for Hadoop.
NOTE: Action failed during Embedded Process processing. Not retrying action due to dataTransferMode setting of parallel.
ERROR: State of connection does not allow data reception.
ERROR: The action stopped due to errors.
NOTE: The Cloud Analytic Services server processed the request in 18.599082 seconds.
quit;
NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE CASUTIL used (Total process time):
real time 18.74 seconds
cpu time 0.00 seconds
dataTransferMode="parallel"
hadoopconfigdir="/opt/sas/viya/config/data/hadoop/conf"
hadoopjarpath="/opt/sas/viya/config/data/hadoop/lib");
NOTE: 'HIVE01' is now the active caslib.
NOTE: Cloud Analytic Services added the caslib 'HIVE01'.
NOTE: Action to ADD caslib HIVE01 completed for session MYSESSION.
/* 100000 rows */
proc casutil;
NOTE: The UUID 'fd16bf47-3bd3-4e44-8eef-85dce5bb9537' is connected using session MYSESSION.
load casdata= "100krows" casout= "100krows"
incaslib="hive01" outcaslib="hive01" copies=0 replace;
NOTE: Performing parallel LoadTable action using SAS Data Connect Accelerator for Hadoop.
ERROR: The Embedded Process job has failed. Please see job log for details. Job tracking URL :
http://server.example.com:8088/proxy/application_1515449747420_0026/.
NOTE: Action failed during Embedded Process processing. Not retrying action due to dataTransferMode setting of parallel.
ERROR: An error occurred executing SAS Embedded Process job.
ERROR: The action stopped due to errors.
NOTE: The Cloud Analytic Services server processed the request in 39.89334 seconds.
quit;
NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE CASUTIL used (Total process time):
real time 39.90 seconds
cpu time 0.00 seconds
/* 1000 rows */
proc casutil;
NOTE: The UUID 'fd16bf47-3bd3-4e44-8eef-85dce5bb9537' is connected using session MYSESSION.
load casdata= "1krows" casout= "1krows"
incaslib="hive01" outcaslib="hive01" copies=0 replace;
NOTE: Performing parallel LoadTable action using SAS Data Connect Accelerator for Hadoop.
NOTE: Action failed during Embedded Process processing. Not retrying action due to dataTransferMode setting of parallel.
ERROR: State of connection does not allow data reception.
ERROR: The action stopped due to errors.
NOTE: The Cloud Analytic Services server processed the request in 18.599082 seconds.
quit;
NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE CASUTIL used (Total process time):
real time 18.74 seconds
cpu time 0.00 seconds
In scenarios where the SAS log reports an Embedded Process job failure (such as in the CASUTIL example above), the YARN application logs might contain the following errors in the stdout log:
WARNING: [01S02]Current catalog set to SASEP (0x80fff8bd)
NOTE: The tcpSockWriteVector call failed. The system error is 'The connection was reset by a peer.'.
ERROR: Socket write vector failed with code 104.
ERROR: tscass3/transmit: Send Msg failed: stmt = 7f4441a9b580 conn = 7f443fc6cf60 port = 4294939625 rc = 8BFFD007
ERROR: Grid BulkOperations error
ERROR: The tcpSockWrite call failed. The system error is 'Broken pipe'.
ERROR: Socket write failed with code 32.
ERROR: Embedded Process DS2 execution instance completed with ERROR [-2130708478].
ERROR: Unable to add rows to table.
NOTE: The tcpSockWriteVector call failed. The system error is 'The connection was reset by a peer.'.
ERROR: Socket write vector failed with code 104.
ERROR: tscass3/transmit: Send Msg failed: stmt = 7f4441a9b580 conn = 7f443fc6cf60 port = 4294939625 rc = 8BFFD007
ERROR: Grid BulkOperations error
ERROR: The tcpSockWrite call failed. The system error is 'Broken pipe'.
ERROR: Socket write failed with code 32.
ERROR: Embedded Process DS2 execution instance completed with ERROR [-2130708478].
ERROR: Unable to add rows to table.
The CAS controller log might contain messages similar to the following:
ERROR [00000007] cas localhost 3301 exampleUser 40852 [tkcasdfxm_hd.c:902] - /sas/wky/mva-vb010/tkcas/src/tkcasdfxm_hd.c:902 HD_runTheDataFeeder: hdepClip->waitForCompletion failed
ERROR [00000007] cas local 3301 exampleUser 40852 [tkcasdf_util.c:1381] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:1381 ReadRows: FAILURE, (0x897FF01C)
ERROR [00000007] cas local 3301 exampleUser 40852 [tkcasdf_util.c:698] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:698 Deleted write cookie (0x897FF01C)
ERROR [00000007] cas local 3301 exampleUser 40852 [tkcasdf_util.c:701] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:701 LoadDFTable: FAILURE, (0x897FF01C)
INFO [00000007] cas local 3301 exampleUser 40852 [tkcasaimp.c:5963] - -- 'loadtable' FAILED. An error occurred executing SAS Embedded Process job.
ERROR [00000007] cas local 3301 exampleUser 40852 [tkcasdf_util.c:1381] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:1381 ReadRows: FAILURE, (0x8BFFD015)
ERROR [00000007] cas local 3301 exampleUser 40852 [tkcasdf_util.c:698] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:698 Deleted write cookie (0x8BFFD015)
ERROR [00000007] cas local 3301 exampleUser 40852 [tkcasdf_util.c:701] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:701 LoadDFTable: FAILURE, (0x8BFFD015)
INFO [00000007] cas local 3301 exampleUser 40852 [tkcasaimp.c:5963] - -- 'loadtable' FAILED. State of connection does not allow data reception.
ERROR [00000007] cas local 3301 exampleUser 40852 [tkcasdf_util.c:1381] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:1381 ReadRows: FAILURE, (0x897FF01C)
ERROR [00000007] cas local 3301 exampleUser 40852 [tkcasdf_util.c:698] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:698 Deleted write cookie (0x897FF01C)
ERROR [00000007] cas local 3301 exampleUser 40852 [tkcasdf_util.c:701] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:701 LoadDFTable: FAILURE, (0x897FF01C)
INFO [00000007] cas local 3301 exampleUser 40852 [tkcasaimp.c:5963] - -- 'loadtable' FAILED. An error occurred executing SAS Embedded Process job.
ERROR [00000007] cas local 3301 exampleUser 40852 [tkcasdf_util.c:1381] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:1381 ReadRows: FAILURE, (0x8BFFD015)
ERROR [00000007] cas local 3301 exampleUser 40852 [tkcasdf_util.c:698] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:698 Deleted write cookie (0x8BFFD015)
ERROR [00000007] cas local 3301 exampleUser 40852 [tkcasdf_util.c:701] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:701 LoadDFTable: FAILURE, (0x8BFFD015)
INFO [00000007] cas local 3301 exampleUser 40852 [tkcasaimp.c:5963] - -- 'loadtable' FAILED. State of connection does not allow data reception.
The CAS worker node logs might contain messages similar to the following:
ERROR [00000006] cas local 12733 exampleUser 40852 [tkcasdf_com.c:5033] - tkcasdf/tkcasdf_com/loadSegmentsForRow: invalid segment 0 found in indicator field. FirstSeg 0, lastSeg 0, bufferChain 1
ERROR [00000006] cas local 12733 exampleUser 40852 [tkcasdf_com.c:5149] - tkcasdf/tkcasdf_com/loadSegmentsForRow: TCPMConnRecvMsgData failed: 0x8BFFD015; tcpstatus=0x0, tcperrno=0x0, tcpflags=0x0
ERROR [00000006] cas local 12733 exampleUser 40852 [tkcasdf_com.c:5150] - State of connection does not allow data reception.
WARN [00000006] cas local 12733 exampleUser 40852 [tkcasdf_com.c:8227] - tkcasdf_com/LoadSCThread(8227): ReadNAdd error. SendStatus: 0x0, RecvStatus: 0x0, SendErrNum: 0, RecvErrNum: 0, Flags: 0x0
ERROR [00000006] cas local 12733 exampleUser 40852 [tkcasdf_com.c:8547] - tkcasdf_com/LoadSCData(8547): A Loader thread has failed (0x8BFFD015)
ERROR [00000006] cas local 12733 exampleUser 40852 [tkcasdf_com.c:8548] - State of connection does not allow data reception.
ERROR [00000006] cas localhost 12733 exampleUser 40852 [tkcasdf_util.c:1381] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:1381 ReadRows: FAILURE, (0x897FF01C)
ERROR [00000006] cas local 12733 exampleUser 40852 [tkcasdf_util.c:698] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:698 Deleted write cookie (0x897FF01C)
ERROR [00000006] cas local 12733 exampleUser 40852 [tkcasdf_util.c:701] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:701 LoadDFTable: FAILURE, (0x897FF01C)
ERROR [00000006] cas local 12733 exampleUser 40852 [tkcasdf_com.c:5149] - tkcasdf/tkcasdf_com/loadSegmentsForRow: TCPMConnRecvMsgData failed: 0x8BFFD015; tcpstatus=0x0, tcperrno=0x0, tcpflags=0x0
ERROR [00000006] cas local 12733 exampleUser 40852 [tkcasdf_com.c:5150] - State of connection does not allow data reception.
WARN [00000006] cas local 12733 exampleUser 40852 [tkcasdf_com.c:8227] - tkcasdf_com/LoadSCThread(8227): ReadNAdd error. SendStatus: 0x0, RecvStatus: 0x0, SendErrNum: 0, RecvErrNum: 0, Flags: 0x0
ERROR [00000006] cas local 12733 exampleUser 40852 [tkcasdf_com.c:8547] - tkcasdf_com/LoadSCData(8547): A Loader thread has failed (0x8BFFD015)
ERROR [00000006] cas local 12733 exampleUser 40852 [tkcasdf_com.c:8548] - State of connection does not allow data reception.
ERROR [00000006] cas localhost 12733 exampleUser 40852 [tkcasdf_util.c:1381] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:1381 ReadRows: FAILURE, (0x897FF01C)
ERROR [00000006] cas local 12733 exampleUser 40852 [tkcasdf_util.c:698] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:698 Deleted write cookie (0x897FF01C)
ERROR [00000006] cas local 12733 exampleUser 40852 [tkcasdf_util.c:701] - /sas/wky/mva-vb010/tkcas/src/tkcasdf_util.c:701 LoadDFTable: FAILURE, (0x897FF01C)
Click the Hot Fix tab in this note to access the hot fix for this issue.
Operating System and Release Information
| Product Family | Product | System | Product Release | SAS Release | ||
| Reported | Fixed* | Reported | Fixed* | |||
| SAS System | SAS Viya | Linux for x64 | 3.3 | 3.4 | Viya | |
An update for this issue is available for SAS Viya 3.3. For instructions on how to access and apply software updates, see the Managing Your Software section of the SAS Viya 3.3 Administration document at
http://documentation.sas.com/?softwareId=administration&softwareVersion=3.3&softwareContextId=softwareUpdates
This SAS Note describes a failure of the parallel data transfer mode of the LOAD statement in PROC CASUTIL from Hadoop CASLIBs.
| Type: | Problem Note |
| Priority: | high |
| Date Modified: | 2018-02-07 13:30:06 |
| Date Created: | 2018-02-05 11:31:29 |