Batch Job Scheduling and Dependencies

Batch job schedulers are often viewed as simply triggering jobs based on calendar or time events, e.g., run monthly reports at the end of the month at 10PM. But job schedulers need to support many other triggering events (i.e., dependencies) beyond calendar or time-based events.

Consider this common case: run customer reports at the end of the business day. Sounds simple, but what does “end of business day” really mean? For some, it’s simply a time; such as 11:45 PM. But in most instances, the “end of business day” is a collection of events which in their entirety announce “end of business day.” Business processes are complete, database backups are performed, archives are stored, emails of end of day notification sent, and only then has the “end of business day” really occurred.

To satisfy such a triggering event, a batch job scheduler may need to, for example:

  • Wait for one or more calendar events to occur
  • Wait for the presence of one or more files
  • Detect the lack of presence of one or more files
  • Detect a database condition or queue condition or an incoming email
  • Await an incoming web service request
  • Wait for user input within a workflow to determine what path the workflow should take based on information collected within the workflow
  • Wait for other workflows to complete entirely or complete to a given step
  • Abort or skip processing within a workflow based on results gathered elsewhere in the workflow, such as when the workflow executes multiple and parallel flows within itself and one flow realizes that another flow needs to be skipped or aborted
  • Wait for the availability of a remote server or agent to perform some processing
  • In a workflow that performs parallel processes, wait for multiple paths within the workflow to complete and join before continuing

Beyond the above triggering events, job schedulers should also provide triggers when a process errors out and times out, or when the workflow is approaching an SLA deadline or has exceeded and SLA deadline.

Combining these events can create powerful and robust workflows that address a wide variety of business cases. Consider the following case, that based on a time event, the workflow searches for files to process them, transfers, summarizes, and records the transfer into a database table. Red lines indicate error flows that track and initiate error processing for a failed transfer.

flux batch job scheduling workflow with dependencies

Or in this case, a workflow is used to announce when it is to run based on the anticipation of some other workflow completing on time (satisfying some audit event). If the other workflow does not complete on time, a message will be shown in the console letting the operator know that the prerequisite workflow is late in completing and that the current workflow will retry in 30 seconds. This workflow itself is used to start another separate workflow (via the flowchart action) so that the started workflow can have its initiation separated from its processing. Separating workflows into those that initiate work versus those that process work can greatly enhance the reusability and flexibility of workflows.

flux batch job scheduling workflow with dependencies

In summary, the availability of many triggering events (aka “dependencies”) within a batch job scheduler can dramatically increase the number of use cases that a batch job scheduler can address. Thoughtful use of dependencies and triggering events can reduce the overall number of workflows and improve their maintainability and flexibility while addressing an ever-increasing scope of your batch scheduling business cases.

To explore batch job scheduling and dependencies further, please contact our team at support@flux.ly or call +1 (720) 441-1844.