The DS2 procedure can run thread programs inside Hadoop when one of the following settings is used:

• The DS2ACCEL system option is set to ANY.
• The DS2ACCEL system option is set to YES in the PROC DS2 statement.

However, Apache Hive 0.13 and later no longer allows any periods (".") in quoted column names, which disallows the use of a period by PROC DS2 as a delimiter within FIRST.variable and LAST.variable flags. This new Hive restriction affects any MERGE and multitable SET statements with a BY statement that are executed in Hadoop. Single-table SET statements are not affected.

When this issue occurs, the SAS^® log can display a combination of messages that are similar to the following:

This issue applies to the following Hadoop distributions:

• MapR Converged Data Platform 5.2 or 6.0
• Hortonworks Data Platform 3.0
• Cloudera Data Platform 5.2 and newer

It can occur with Amazon Web Services (when a Hive version of 0.13 or later is selected).

Click the Hot Fix tab in this note to access the hot fix for this issue.

Operating System and Release Information

/* You can specify the DS2ACCEL option on an OPTIONS statement */

options ds2accel=any;

/* specify the appropriate path and options for your Hadoop connection */
 
libname myhdp hadoop server='server123.unx.yours.com';

/* You can specify the DS2ACCEL option on the PROC statement */

proc ds2 ds2accel=yes;
   thread t_pgm /overwrite=yes;
     method run();
       merge myhdp.a myhdp.b;
       by x y;
     end;
   endthread;
run;

data nyhdp.new/overwrite=yes;
   dcl thread t_pgm t;
     method run();
     set from t;
   end;
enddata;
run;
quit;

A fix for this issue for SAS Threaded Kernel DS2 and FedSQL Language Processors 9.41_M3 is available at: