DataFlux Data Management Studio 2.7: User Guide

Tokenization Node

The Tokenization Node represents the tokenization processing carried out by a definition. This is necessary for definitions that treat individual tokens differently, to determine which substrings in an input string correspond to the tokens.

When the tokenization definition specified has no solutions found, there will be no output for the Tokenization node or for any of the groups or nodes within the token groups that follow, except when the Tokenization node is used in a Match definition.

Used in:

• Gender Analysis Definitions
• Match Definitions
• Standardization Definitions

Input Properties Within Gender and Standardization Definitions

Parse definition

This is the name of the parse definition. All available parse definitions that belong to the same data type are shown here.

Allow input that is already parsed

If this check box is selected, you can choose to allow previously-parsed input as well as a single string. This parsed input can come from external sources (for example, columns of a database table) or from a previous run of the parse definition. No matter where the data comes from, the input fields must correspond to the tokens on the selected parse definition.

Input Properties Within Match Definitions

Definition type:

Enables you to select the type of definition that should be used. The choices are: (None), Parse and Extraction.

Definition:

Enables you to select the definition that should be used.  Click Open definition to view and edit the selected definition's Flow diagram.

Allow input that is already tokenized

If this check box is selected, you can choose to allow previously-tokenized input as well as a single string. This tokenized input can come from external sources (for example, columns of a database table) or from a previous run of the definition. No matter where the data comes from, the input fields must correspond to the tokens on the selected definition.

Output Properties

The output of the definition, if a single string was input. If parsed input was used, the parsing step is skipped, so there is no output.