flux support for amazon aws s3

Flux Support for AWS S3

With its latest release Flux adds Amazon AWS S3 support to its list of supported file transfers. With these new Flux actions users can interact with their S3 instance to manage their buckets and objects. Using Flux provides a more practical and cost effective substitution to support a single point of control for your file management. Flux’s S3 actions move …

developer handling late scheduling of jobs

I’m Late, I’m Late. Handling the Late Scheduling of Jobs

A scheduled batch job might not run on time (i.e., be “late”) for many reasons – for example: Such circumstances can often be handled manually by pausing or restarting jobs, but this can quickly become overwhelming, especially when many jobs are involved. To handle this, jobs need a defined way of telling the scheduler what the absolute latest time the …

software developer date and time scheduling

Dates, Times, and Calendars – Oh My!

Workload automation projects generally start simple. The first step is often setting up the schedule of when to run jobs. Schedules can start out straightforward, but over time the kinds and types of schedules needed to be addressed evolve and grow increasingly complex. A commonly used time expression language is that of Unix Cron. We here at Flux have extended …

Batch Scheduling Multiple Files

When utilizing a batch job scheduler like Flux, how does one process multiple files concurrently?  Many batch schedulers execute workflows that process a file one at a time and from beginning to end. These files are scheduled to run through a workflow while many are ready to process. Given this scenario, files begin to back up as they wait for …

Batch Job Scheduling and Managed File Transfers

Transferring files using a batch job scheduler with fine-grained control and configuration at runtime is a key and often used Flux feature. The following describes one such Flux workflow developed for a large mortgage broker in their transfer and tracking of mortgage files. The same model is in use in insurance, invoicing, logistics, and many other industries needing flexible and …

Batch Job Scheduling and Dependencies

Batch job schedulers are often viewed as simply triggering jobs based on calendar or time events, e.g., run monthly reports at the end of the month at 10PM. But job schedulers need to support many other triggering events (i.e., dependencies) beyond calendar or time-based events. Consider this common case: run customer reports at the end of the business day. Sounds …

Provisioning SaaS Providers with Customer Data Feeds

Batch Job Schedulers: Doing More Than Simply Scheduling From fraud detection to payment clearing and digital media translation, many SaaS providers bring additional value for batch scheduled data feeds provided by their customers. They also use a wide range of methods for data collection and transfer from their customers – often slow, error-prone, and difficult to centrally monitor the data.  …

Batch Job Orchestration: Batch Job Scheduling + File Processing

Batch Job Scheduling Batch job scheduling is a key and often overlooked facility of modern enterprise systems. Although considered ‘low-tech’ in this age of streaming video, mobile, big data, and high-speed messaging, much of an enterprise’s day to day processing taking place today is via scheduled batch jobs. Batch job scheduling is frequently addressed in a fragmented, siloed, application-by-application basis. …

Making Reusable File Transfer Workflows

How does Flux make file transfer workflows easier? A Frequent Question An example of a frequent query (edited here for clarity) received by our technical support staff: “… we have a lot of file transfers that all implement the same logic. So we’d like to design this logic once and then just feed the transfer details as parameters for runtime …