Categorizing Batch Schedules

There exist many kinds of batch schedules. These schedules may vary in complexity but can still share a high degree of commonality. Simply categorizing your batch schedules affords insight into this commonality and offers opportunity to simplify or find other scheduling opportunities. Some batch schedule categories are provided below as candidate guidance to jumpstart organizing your enterprise’s batch schedules.

1. Run

This is the simplest category of job schedules. While simple, this case is very common. The schedule involves scheduling a command line (e.g., a MySQL or SQL Server database backup), shell script, batch script, DOS or PowerShell command to run periodically or on-demand.

2. Run and Route

This schedule category involves a periodic or adhoc run and then routing the results to a target destination. A destination could be email, into a database stored procedure, or to a remote file server via a file transfer protocol such as FTP or SFTP.

3. Retrieve, Review, Reject, Repair, and Route

Another schedule category involves retrieving files from local or remote sources and then reviewing their contents, possibly rejecting or repairing the files, and then routing the result to destination processes such as database loads, an ERP system, or even machine learning processes. Flux excels in these cases since Flux integrates file transfers with batch job scheduling.

A particular schedule may look like this:

  • On a periodic basis check a remote or local directory for the presence of a file.
  • Retrieve (i.e., copy or move) that file securely from the source location to a target destination.
  • Review the file to ensure necessary data is present. Maybe the file needs to be a minimum size, or have a certain number or records. The exact process will vary depending upon your requirements.
  • Reject duplicate files, files that are too small, or files that are determined not to be repairable. Possibly send a notification to the file’s creator of what the issues with the file are.
  • If the file is determined to be repairable, route the file to an automated or manual repair process.
  • Once the file is repaired, retrieve and then route the file to its eventual destination.

4. Retrieve, Review, Reject, Repair, Reformat, and Route

An extension of the schedule category #3 is to inject a reformat step into the process. The retrieved files may, for example, need to be translated into different languages or converted into a different format before the file is uploaded to the target destination.

In Summary

A surprisingly large number of batch schedules can be automated using the above four schedule categories as templates. Many variants of the above categories also exist. The above are presented simply as a starting point. We used alliteration with the letter “R” simply to make them easier to remember.

Note that the above categories mention some time of clock event to trigger the start of the process but many other kinds of triggers may be employed to initiate the above cases. That topic will be dealt with at a later point. Stay tuned.