Call Sales +1 (720) 930-4239 | Email Sales sales@flux.ly — Call Support +1 (720) 441-1844 | Email Support support@flux.ly

Job Scheduling and File Transfer

Batch Scheduling Multiple Files

flux batch schedule multiple files

When utilizing a batch job scheduler like Flux, how does one process multiple files concurrently? 

Many batch schedulers execute workflows that process a file one at a time and from beginning to end. These files are scheduled to run through a workflow while many are ready to process. Given this scenario, files begin to back up as they wait for the prior file to finish and proceed with the next file through the workflow. 

But if one separates and categorizes workflows between those that listen for incoming work, schedule that work, and those that then perform the work, new opportunities arise.

Let’s look closer at a scenario used by many payment vendors concurrently processing incoming payment files to post on their back-end systems. A “listening workflow” listens for files and moves them to what can be called the landing zone. A “scheduling workflow” waits for files to arrive into the landing zone. Upon arrival of the files, a processing workflow initiates processing for each file.

Within the listening workflow, a File Move action moves files to the landing zone. The scheduling workflow watches for these moved files. The scheduling workflow then creates the processing workflow and passes the filename to this workflow for processing. When creating the processing workflow, the filename or some metadata is extracted from the file and is appended to the workflow’s name – uniquely identifying what file is being processed for graphical monitors or a dashboard.

The following Flux workflow outlines this process graphically. The Initiate Process per File starts many processing workflows to handle the concurrent processing of files. These processing workflows can be as complex as required to process the file and are not shown in detail here.

flux workflow example for scheduling multiple batch files

Such reusable processes are an integral feature of Flux.

Contact sales@flux.ly for additional examples or further details.

Related posts

Getting to Done Quickly

Data collection made easy for global healthcare provider using Flux With Growth Comes Added...

Can we help? Tell us.

Share with us the contact information for a friend or fellow developer (or even...

Flux or Quartz

Looking for a Java job scheduler? After some searching and introspection, perhaps you arrive...

The Origins of Flux

Sims Computing created Flux, an enterprise component, using Java Technology Sims Computing was founded...
Top