DataFlux Data Management Studio 2.6: User Guide

Exporting Lineage Data to SAS Web Infrastructure Platform

Overview

You can export lineage data from the active repository for a DataFlux Data Management Server to the SAS Web Infrastructure Platform (WIP). Clients such as SAS Lineage Viewer and SAS Business Data Network can then use this lineage data in various ways.

Perform the following tasks to export lineage data from a DataFlux Data Management Server to WIP: 

• Copy the Export Lineage Job
• Perform a Test Run of the pjExportLineage Job
• Schedule and Run the Job

Copy the Export Lineage Job

When a DataFlux Data Management Server is installed, the job that supports lineage export is installed in a backup folder. In order to export lineage, you must copy the job that supports it from its back up folder to the folders where the server expects it.

• Copy FROM dmserver_home\share\web\batch jobs\Lineage TO dmserver_home\var\batch jobs\lineage

A script is provided to move these jobs to the correct location. Under Windows, click the shortcut dmserver_home\Update Web Studio Jobs. Under UNIX, run dmserver_home/bin/update_jobs.

Perform a Test Run of the pjExportLineage Job

The pjExportLineage job is installed with the DataFlux Data Management Server. This job calculates the lineage for the active repository for the server and uploads the results to the SAS Web Infrastructure Platform (WIP). Note that you should never need to edit or even open this job.

You should perform a test run of the job before you schedule and run it to ensure that the lineage data is properly exported to WIP and the data can be viewed in SAS Lineage Viewer. Perform the following steps:

1. Open the Data Management Servers riser.
2. Open the folder for the DataFlux Data Management Server that contains the repository that contains the lineage that you need to export.
3. Log on to the server, if prompted.
4. Navigate to the Batch Jobs folder. Then, open the lineage folder.
5. Right-click pjExportLineage and click Run in the pop-up menu to access the Test Real-Time Process Service window.
6. Enter the variables listed in the following table into the appropriate rows in the table in the Variable Inputs and Outputs tab:

   Variable	Description
   DEBUG_FILE (optional)	The location of the file that you would like to store debugging information. If this value is not set, then debugging information will not be output. Example value: c:\DM_Server_ExpLineage\debug.txt
   TMPDIR (optional)	The storage location of the lineage XML export.  Example value: c:\DM_Server_ExpLineage\tempdir
   EXPORT_USERNAME (required)	The Base64-encoded representation of the user ID for the WIP user of the import service, as defined in the SAS Metadata Server. Example value: ZWRpdGJkbg==
   EXPORT_PASSWORD (required)	The Base64-encoded representation of the password for the WIP user of the import service. Example value: T3Jb24xMjM
   EXPORT_URI (required)	The URI to the import service, which is WIP. URI Syntax: http:/myWIPserver.com:port_number/SASLineage/rest/legacy/ Where: myWIPserver.com is your WIP server name. port_number is the port number for SAS Lineage Viewer on the WIP server. You do not need to specify the port number for SAS Lineage Viewer if the default port was accepted during installation. /SASLineage/rest/legacy/ is the literal path for SAS Lineage Viewer on the WIP server. Note the trailing slash.
   EXPORT_HTTP_CONFIG_FILE	The HTTP configuration file for timeout values as needed. When uncommmented, enables overriding HTTP options for web service nodes in the export job. Example value: C:\Program Files\DataFlux\DMServer\server1\etc\ws-lineage.xml

   Note that the pjExportLineage job expects all user names and passwords to be Base64 encoded to prevent plain text credentials from being saved and exposed. You can encode your credentials with the following PERL command:

   perl -e "use MIME::Base64; print encode_base64("your user name or password here");"

   If you use a Windows machine that does not have access to PERL, note that you can find Internet websites that will accept a text string and return a Base64 encoded string.

7. Click Run. This test run actually exports the lineage data from the DataFlux Data Management Server to WIP.
8. Use SAS Lineage Viewer to display the exported lineage information. For instructions, see the documentation for SAS Lineage Viewer.

Schedule and Run the Job

You can schedule and run the pjExportLineage job in the scheduling application of your choice. The run command uses the following pattern:

path_to/dmpexec -j path_to/pjExportLineage.djf -i DEBUG_FILE=...-i TMPDIR=...-i EXPORT_USERNAME=- -i EXPORT_PASSWORD= -I EXPORT_URI=-o BASE/REPOS_FILE_ROOT=)

The variables that you enter are the same as those listed in the table in the Test the pjExportLineage Job section (except -o BASE/REPOS_FILE_ROOT=). For information about using the dmpexec command to run the job, see Running Jobs from the Command Line.

Note that you can also set variables in the macros file to correspond to these variables. Then you can simply add the macros file to the command with the -j filename option to the dmpexec command. For general information about using macro variables, see Using Macro Variables. The macro variables that correspond to the pjExportLineage job variables are listed in the following table.

The job will export the data to the WIP location specified in the EXPORT_URI variable or the LINEAGE/EXPORT_URI macro variable. See the documentation for SAS Lineage Viewer for information about reviewing and using the lineage data.