The HPCOUNTREG Procedure

This example shows the use of the HPCOUNTREG procedure with an emphasis on large data set processing and the performance improvements that are achieved by executing in the high-performance distributed environment.

The following DATA step generates one million replicates from the zero-inflated Poisson (ZIP) model. The model contains seven variables and three variables that correspond to the zero-inflated process.

    data simulate;
       call streaminit(12345);
       array vars x1-x7;
       array zero_vars z1-z3;

       array parms{7}  (.3 .4 .2 .4 -.3 -.5 -.3);
       array zero_parms{3} (-.6 .3 .2);

       intercept=2;
       z_intercept=-1;
       theta=0.5;

       do i=1 to 1000000;
          sum_xb=0;
          sum_gz=0;
          do j=1 to 7;
             vars[j]=rand('NORMAL',0,1);
             sum_xb=sum_xb+parms[j]*vars[j];
          end;
          mu=exp(intercept+sum_xb);
          y_p=rand('POISSON', mu);

          do j=1 to 3;
             zero_vars[j]=rand('NORMAL',0,1);
             sum_gz = sum_gz+zero_parms[j]*zero_vars[j];
          end;
          z_gamma = z_intercept+sum_gz;
          pzero = cdf('LOGISTIC',z_gamma);
          cut=rand('UNIFORM');
          if cut<pzero then y_p=0;
          output;
       end;
    keep y_p x1-x7 z1-z3;
    run;

The following statements estimate a zero-inflated Poisson model.

 option set=GRIDHOST="&GRIDHOST";
 option set=GRIDINSTALLLOC="&GRIDINSTALLLOC";

 proc hpcountreg data=simulate dist=zip;
    performance nthreads=2 nodes=1 details
            host="&GRIDHOST" install="&GRIDINSTALLLOC";
    model y_p=x1-x7;
    zeromodel y_p ~ z1-z3;
 run;

The model is executed in the distributed computing environment on two threads and only one node. These settings are used to obtain a hypothetical environment that might resemble running the HPCOUNTREG procedure on a desktop workstation with a dual-core CPU. To run these statements successfully, you need to set the macro variables GRIDHOST and GRIDINSTALLLOC to resolve to appropriate values, or you can replace the references to the macro variables in the example with the appropriate values. Output 6.1.1 shows the "Performance Information" table for this hypothetical scenario.

Output 6.1.2 shows the results for the zero-inflated Poisson model. The "Model Fit Summary" table shows detailed information about the model and indicates that all one million observations were used to fit the model. All parameter estimates in the "Parameter Estimates" table are highly significant and correspond to their theoretical values set during the data generating process. The optimization of the model that contains one million observations took 40.77 seconds.

In the following statements, the PERFORMANCE statement is modified to use a grid with 10 nodes, with each node capable of spawning eight threads:

 proc hpcountreg data=simulate dist=zip;
    performance nthreads=8 nodes=10 details
                host="&GRIDHOST" install="&GRIDINSTALLLOC";
    model y_p=x1-x7;
    zeromodel y_p ~ z1-z3;
 run;

Because the two models being estimated are identical, it is reasonable to expect that Output 6.1.2 and Output 6.1.3 would show the same results. However, you can see a significant difference in performance between the two models. The second model, which was run on a grid that used 10 nodes with eight threads each, took only 3.54 seconds instead of 40.77 seconds to optimize.

In certain circumstances, you might observe slight numerical differences in the results, depending on the number of nodes and threads involved. This happens because the order in which partial results are accumulated can make a difference in the final result, owing to the limits of numerical precision and the propagation of error in numerical computations.

As this example suggests, increasing the number of nodes and the number of threads per node improves performance significantly. When you use the parallelism afforded by a high-performance distributed environment, you can see an even more dramatic reduction in the time required for the optimization as the number of observations in the data set increases. When the data set is extremely large, the computations might not even be possible in some cases, given the typical memory resources and computational constraints of a desktop computer. Under such circumstances the high-performance distributed environment becomes a necessity.