Using Scheduling to Handle Complex Process Flows

Problem

You have a complex job involving joins and transformations from many different tables. You want to reduce the complexity by creating a set of smaller jobs that are joined together and can then be executed in a particular sequence.

Solution

Group all of the jobs in the flow together in a single folder in the Folders tree. Perform the steps in Schedule Complex Process Flows to deploy and schedule the jobs in the proper sequence.
As an alternative to the approach described here, you can drop jobs into other jobs and build up complexity that way. For example, you can build an outer job that contains inner jobs. You might find that these nested jobs provide a more direct and efficient solution to the problem of creating and scheduling complex process flows. This approach does not require separate deployment steps. For more information, see Creating a Job That Contains Jobs.

Tasks

Schedule Complex Process Flows

Perform the following steps to schedule complex process flows:
  1. Divide the complex job into a series of smaller jobs that create permanent tables. Those tables can then be used as input for succeeding jobs.
  2. Keep all of your jobs in the flow together in a single folder in the Folders tree, and give the jobs a prefix that displays them in the appropriate execution order.
  3. Deploy the jobs for scheduling.
  4. The user responsible for scheduling can use the appropriate software to schedule the jobs to be executed in the proper sequence.