SPD Server with Hadoop Configuration Examples

Using the libnames.parm File

One way to access a Hadoop cluster (without Kerberos) is to specify a domain in the libnames.parm file, and then add the HADOOPCFG= and HADOOPJAR= parameter file options.

Here is an example of the libnames.parm file that specifies the HADOOPCFG= and HADOOPJAR= parameter options:

libname=foo 
   pathname=/user/userlname/mydomain 
   hadoopcfg=/u/hadoop/hdist/cdh/confdir/hdp20p1
   hadoopjar=/u/hadoop/hdist/cdh/sas_cdh20u;

Using the spdsserv.parm File

Another way to access a Hadoop cluster (without Kerberos) is to specify the HADOOPCFG= and HADOOPJAR= options in the spdsserv.parm file. Then, you can specify multiple domains in your libnames.parm file. Each of those domains gets its Hadoop cluster configuration files and Hadoop distribution JAR files from the spdsserv.parm file.

Here is an example of the HADOOPCFG= and HADOOPJAR= parameter options specified in the spdsserv.parm file:

hadoopcfg=/u/hadoop/hdist/cdh/confdir/cdh54d1;
hadoopjar=/u/hadoop/hdist/cdh/cdh54;

Then, in the libnames.parm file, you only need to specify a libref, a Hadoop directory, and the parameter option HADOOP=YES.

libname=Stuff1 pathname=/user/userlname/mydomain1 hadoop=yes;
libname=Stuff2 pathname=/user/userlname/mydomain2 hadoop=yes;

To use the domains that are defined in the libnames.parm file, specify them in the same way as any other SPD Server domain in client code:

libname Stuff1 sasspds ‘Stuff1’ server=lax94d01.14526 user=”anonymous”;