Dates, Times, and Calendars – Oh My!

Workload automation projects generally start simple. The first step is often setting up the schedule of when to run jobs. Schedules can start out straightforward, but over time the kinds and types of schedules needed to be addressed evolve and grow increasingly complex. A commonly used time expression language is that of Unix Cron. We here at Flux have extended …

provisioning flux workflows to maximize utility for a wide variety of uses

Provisioning Workflows to Simplify Batch Schedule Deployments

Designing and deploying batch job schedules can be complex and intimidating. Coordinating dependencies, balancing CPU and disk resources, and tracking down errors can be time-consuming and error-prone. But after design, how do you deploy your workflows to maximize their utility for a wide variety of uses? Workflow provisioning provides a solution. By taking your best practice workflows and embedding variables …

Batch Scheduling Multiple Files

When utilizing a batch job scheduler like Flux, how does one process multiple files concurrently?  Many batch schedulers execute workflows that process a file one at a time and from beginning to end. These files are scheduled to run through a workflow while many are ready to process. Given this scenario, files begin to back up as they wait for …

Batch Job Scheduling and Managed File Transfers

Transferring files using a batch job scheduler with fine-grained control and configuration at runtime is a key and often used Flux feature. The following describes one such Flux workflow developed for a large mortgage broker in their transfer and tracking of mortgage files. The same model is in use in insurance, invoicing, logistics, and many other industries needing flexible and …

Batch Job Scheduling and Dependencies

Batch job schedulers are often viewed as simply triggering jobs based on calendar or time events, e.g., run monthly reports at the end of the month at 10PM. But job schedulers need to support many other triggering events (i.e., dependencies) beyond calendar or time-based events. Consider this common case: run customer reports at the end of the business day. Sounds …

Provisioning SaaS Providers with Customer Data Feeds

Batch Job Schedulers: Doing More Than Simply Scheduling From fraud detection to payment clearing and digital media translation, many SaaS providers bring additional value for batch scheduled data feeds provided by their customers. They also use a wide range of methods for data collection and transfer from their customers – often slow, error-prone, and difficult to centrally monitor the data.  …

Batch Job Orchestration: Batch Job Scheduling + File Processing

Batch Job Scheduling Batch job scheduling is a key and often overlooked facility of modern enterprise systems. Although considered ‘low-tech’ in this age of streaming video, mobile, big data, and high-speed messaging, much of an enterprise’s day to day processing taking place today is via scheduled batch jobs. Batch job scheduling is frequently addressed in a fragmented, siloed, application-by-application basis. …

Making Reusable File Transfer Workflows

How does Flux make file transfer workflows easier? A Frequent Question An example of a frequent query (edited here for clarity) received by our technical support staff: “… we have a lot of file transfers that all implement the same logic. So we’d like to design this logic once and then just feed the transfer details as parameters for runtime …

Digital Catalog Job Scheduling

The number and kinds of batch scheduling workflows are diverse. One valuable category of batch job scheduling activity is the scheduling and processing of digital media into on-line catalogs. The nature of digital media varies widely, from audio files to digital photos and movies to PDF-formatted contracts and legal documents. Frequently this media is processed and loaded into a digital …