Provisioning Workflows to Simplify Batch Schedule Deployments

Designing and deploying batch job schedules can be complex and intimidating. Coordinating dependencies, balancing CPU and disk resources, and tracking down errors can be time-consuming and error-prone.

But after design, how do you deploy your workflows to maximize their utility for a wide variety of uses?

Workflow provisioning provides a solution. By taking your best practice workflows and embedding variables within them that are resolved at runtime, and adding a provisioning workflow to configure and submit your workflows, you can take your workflows and use them repeatedly.

Such a facility requires that your workflow engine or scheduler allows data entry at the time the workflow is submitted for scheduling. The entered data is used to “provision” the workflow with needed values for the variables defined in the workflow.

Say we want a workflow that we will use repeatedly to start other workflows. We want this initiating workflow to provision other workflows with their name, time to run, command to execute, and email addresses for sending success and failure notifications. We will call this initiating workflow “EasyWorkflow.”

First, we add a provision form to our EasyWorkflow workflow. The form itself is defined using Json and will be rendered when we submit EasyWorkflow for execution. A snippet of the Json is illustrated below.

Collect needed workflow values using a Json form definition

The complete version of the above form collects five pieces of information with the following variable names:

  • ENGINENAMESPACE – The namespace of where to run the SimpleCommand workflow
  • TIMEEXPRESSION – A cron expression or relative time expression of when to run
  • COMMAND – A batch script, shell script, or command line to execute
  • FAILEDCONTACT – An email address to notify when the command fails
  • SUCCESSCONTACT – An email address to notify when the command succeeds

EasyWorkflow contains a flow chart action which is used to initiate the other flow chart. This flow chart action is used to fetch another flow chart and inject the values collected from the above form into the flow chart specified (in this case a workflow named SimpleCommand). SimpleCommand will be initiated separate and apart from EasyWorkflow.

The SimpleCommand workflow is shown below. It simply runs a specified command within a process action based on a time expression specified within a timer trigger.

SimpleCommand workflow

The definition for how EasyWorkflow will start SimpleCommand is shown below. EasyWorkflow will submit SimpleCommand for execution with the values from its form by turning on “Initialize from Flow Context”. EasyWorkflow will submit SimpleCommand to the scheduler engine with the variable value with the name ENGINENAMESPACE collected from the form above.

Define what and how to submit a workflow from a repository of stored workflows

When EasyWorkflow is submitted from the repository of workflows to the scheduler for execution, it renders the following form to the submitting user. Default values are provided to be edited for your specific needs. In this case the command is being scheduled to run at 5 AM using the time expression “0 0 0 5” (i.e., the 0th millisecond, 0th seconds, 0th minutes, 5th hour of every day).

The Json form definition drives a web data entry form to simplify the configuring and submission of workflows

Illustrated below: SimpleCommand shows being scheduled on the engine as TestFlow. Execution is scheduled for 5 a.m. EasyWorkflow, after submitting the job, completes. The same EasyWorkflow is used repeatedly to schedule many jobs onto the scheduler with different scheduled times and as different names

Use EasyWorkflow repeatedly to schedule different jobs at different times

EasyWorkflow is a simple example of the power available in combining provisioning with workflows and variable substitution. By extending the data collected within EasyWorkflow and using it to drive workflows (beyond SimpleCommand of course!) one can provision a wide variety of common use case workflows such as file transfers, ETL loads, data analytics runs, and report generations.

For additional examples or further details, contact sales@flux.ly or call +1 (720) 930.4239.