DataFlux Data Management Studio 2.5: User Guide

Token Config Node

The Token Config Node configures the sequential order of processing for tokens.

Used in:

• Gender Analysis Definitions

Properties

Table consists of one row for each token, and two columns:

• Token - name of the token
• Stop - whether to skip any tokens that have not yet been processed, if this token has assigned a gender

Select a row and click the up and down arrows to reposition the row.

Output

None

Notes

Unlike other definitions, the Gender Analysis Definition processes the tokens sequentially, in the order set by you. It also enables you to permit certain tokens' outputs to be considered more definitive than others. For example:

Input: "Mrs John Smith", referring to the wife of John Smith. In this case, "Mrs" is the definitive word.

If all the tokens are processed in the natural order, the outputs for the tokens are:

• Name Prefix - "Mrs" (Female)
• Given Name - "John" (Male)
• Family Name - "Smith" (not used in the definition, since Family Name does not tell us anything about gender)

Since there is one female and one male assignment, the overall final gender assignment is unknown. However, processing stopped after the first token (Name Prefix) that has assigned a gender, the correct result is (Female). If Given Name was processed before Name Prefix, this does not work.