DataFlux Data Management Studio 2.5: User Guide

Managing an HTTP Request

You can create a data job that calls web services with a Web Service node or an HTTP Request node. For example, you can create a job that takes input from a Job Specific Data node, processes it in an HTTP Request node, and sends it to an XML Column Input node. Then, you can use the data in other DataFlux Data Management Studio processes. Perform the following tasks:

• Create an HTTP Request
• Prepare the Data for an XML Column Input Node
• Send the Data to the XML Column Input Node

Note that you can build a similar job for the Web Service node. See also What Can I Do About Time-Out Errors in Data Jobs with the Web Service Node or the HTTP Request Node?

Create an HTTP Request

You can create a data job to create an HTTP request that processes data that you provide. The data flow for one of these jobs contains the following nodes:

• Job Specific
• Concatenate
• HTTP Request

The following display shows the data flow:

The Job Specific node enables you to specify the fields and values that you want to process in the job. The web service used in the job is the Google Distance Matrix, which calculates the distance between two addresses that you supply. This service requires required parameters that specify the origins and destinations expressed as latitude and longitude values that are separated with a pipe character. The two Concatenate nodes in the flow process the data from the Job Specific Data node to make it conform to this required format. The following display shows the first Concatenate node, which processes the origin data:

Finally, you need to configure the HTTP Request node, as shown in the following display:

The Address field contains the URL needed to connect to the Google Address Matrix web service that is used for this job. Note that the URL contains all of the parameters needed to connect to and use the service. Follow this process to complete this field:

1. Look at the web service description to find out how to call it. In general this URL takes a form similar to the following: http://www.website.com/webservice_name&parm1=abc&parm2=def.
2. Build the address field in your HTTP Request node. You must decide which of the parameter values need to be dynamic. For example, you could assume that both parameters should be dynamic. In that case, you would write the address in the node in the following format: http://www.website.com/webservice_name&parm1={parameter1_value}&parm2={parameter2_value}. Note the use of curly brackets, {}, which mark dynamic parameter values.
3. Click on the Discovery button to populate the parameters into the Parameters field. This action interprets the address and recognizes the two parameters: parameter1_value and parameter2_value. Then the user defines whether this value should 1) come from a field that flows into the node (type COLUMN), 2) just be a fixed value (but the value defined in the node properties and not in the URL itself) (type STRING), or 3) should come from a file (type FILE).
4. Review the name, type, and value of each parameter in the list. If you need to edit a parameter, select it in the list and click Edit. The Edit Parameter dialog is shown in the following display:

   

   The parameter type is determined by your selection of a Field (COLUMN), a File (FILE), or Text (STRING) in the Parameter section of the dialog.

The following value is entered in the Address field for this job:

http://maps.googleapis.com/maps/api/distancematrix/xml?origins={origins_latlong}&destinations={destinations_latlong}&mode={mode}&units={units}&sensor=false

Note the following about the contents of the field:

• You can specify any values that need to come from node itself through dynamic replacement by enclosing them in curly brackets, as in {mode}.
• The &mode= part is web service request language for saying that this web service requires a parameter called mode. Note that this is a parameter of the web service. Google has defined that the web service understands this parameter. The {mode} part is DataFlux-specific and tells the node that whenever it calls the web service, it needs to replace {mode} with the value of the node parameter, which is called mode. In this example, mode is a fixed string that resolves to driving.
• The parameter origins_latlong, has a column parameter type. It is dynamically replaced at run-time with the value of one of the fields that flows into the node. In this case, the field is called field_origins_latlong.

Finally, you need to specify the text file that will contain the results of the HTTP request. The text file for this job is named DistanceData.

Prepare the Data for an XML Column Input Node

You must prepare the data in the file output from the HTTP Request node before you can use it in the XML Column Input node. The XML Column Input node in this job consumes the data in the form of an XML map, which you must generate from the SAS XML Mapper.

The following display shows the contents of the file output from the HTTP Request node:

Note that the lines are followed by linefeed characters. The SAS XML Mapper application cannot accept text with linefeed characters. Therefore, you need to perform the following steps:

1. Open the text file generated by the HTTP Request node in a text editor.
2. Delete the linefeed characters from the file.
3. Save the resulting document as an XML document.

The XML document is shown in the following display:

Now you can open the XML document in the SAS XML Mapper. Briefly stated, create the XML map in the XML Mapper by performing the following steps:

1. Open the XML map in the SAS XML Mapper.
2. Change the default map name (SXLEMAP) to a more appropriate name, such as Google_Distance_Matrix_API.
3. Fully expand the XML code in the XML code field on the left side of the XML Mapper.
4. Drag the top-level of the XML code field (DistanceMatrixResponse) to the matching level in the XML map field on the right side of the XML Mapper.
5. Drag all of the child-levels that you want to include in the XML map from the XML code field to the top level of the XML map field (DistanceMatrixResponse).
6. Click Properties for each level in the map and edit the Name field as needed.
7. Click Format for each level in the map and make sure that the column length is long enough to avoid data truncation.
8. Open the Tools menu and click Options. Click Processing and make sure that the Default XML Map version is set to 2.1.
9. Save the completed XML map.

The following display shows a completed map in the SAS XML Mapper:

Send the Data to the XML Column Input Node

Now you can return to the data job and send the data to an XML Column Input node. First, you need to change the output from the HTTP Request node. To do this, open the HTTP Request node properties and specify request_string as the output field. Click Edit to make sure that Maximum length value is long enough to contain all of the data in a single field. This data job uses a value of 1024,.as shown in the following display:

The next step is add an XML Column Input node to the data flow after the HTTP Request node. The node should be configured as shown in the following display:

Note that you need to enter the result_string output field from the HTTP Request node into the XML Input field. You also need to add the XML map and table name from the SAS XML Mapper into the appropriate fields. Now you can preview the XML Column Input node to see the results, as shown in the following display:

You can see that the fields that were added to the XML map are shown as columns in the Preview tab. They will be inserted into the XML Column Input node when you run and successfully complete the data job.