Automation for Changing Workloads

Flux and ETL

One of the Use Cases for Flux

Flux is used to create and execute automated workflows that involve a combination of steps and tasks across servers. Flux can initiate time-based schedules, batch jobs through scripting, interface with databases through SQL, and MFT (managed file transfer) activities through support of file transfer protocols such as FTP and SSH-based FTP. With many customers in the financial services industry, Flux is often deployed to help automate ETL (extract, transform and load) and business intelligence tasks.

The tasks that Flux has been asked to help automate have changed over the years. In today’s enterprise automating batch jobs at 3 a.m. is no longer the priority.

Some History

In the early days that used to be the most common use case or problem that Flux solved. It still happens. But today, it’s evolved to focus more on MFT. At 3 a.m., maybe you’ll start monitoring your customers’ FTP servers. Or maybe you’ll call out to this Web service, and other kinds of SOA [service oriented architecture] environments.

Integrating with third-party services made available over the Web is also becoming more popular than exclusively coordinating a company’s internal servers and job processes. We are seeing less and less of ‘Let’s script this process,’ and more of ‘Let’s integrate it with this other service that’s been made available across the network.’ A lot of key IT infrastructure you see today is distributed across different companies.

A Typical Example

A financial services customers uses Flux’s Web service APIs to access an external anti-fraud service. The vendor who supplies that fraud service does not package that piece of software inside the firewall. It’s only available as a service. There are obvious advantages to that. But the question becomes, how are you, as a company and as an IT department, going to orchestrate and automate all of these things together, with services that are no longer inside your firewall, but out there on the Internet? Flux supports Web services and SOA. Flux provides a RESTful API as a way of integrating with external Web services. Flux also supports for SOAP.

Putting IT into Centralized Control

Flux’s goal is to put the IT professional into a Web browser, where he or she does everything. Just about all user interaction in Flux occurs in a Web browser, including the drag-and-drop design tool, which allows users to create jobs and workflows–all with various dependencies and logic.

All Flux activity is centrally monitored from a browser-based console.For example, Flux provides highly granular access controls to the its Operations Console. An administrator setting up Flux will expose only the components of the tool that users need to do their jobs, and conceal the parts they don’t need. Flux also handles the workflow promotion lifecycle, from development to testing and production.

Many Flux customers use Flux to automate ETL processes for loading data warehouses. File transfers go hand in hand with business intelligence and ETL. The business intelligence ETL applications need data to work on, and Flux is often used to get the data there.

Flux is Flexible and Scalable

Flux gives customers the option of deploying a Java-based Flux engine on each server involved in a workflow, or deploying a small agent to the targeted server. Most customers chose the full engine approach, according to Sims, who considers the agent-less approach and use of a single Java code base as competitive advantages over other workload automation and MFT vendors.

With Flux, you can roll out almost as many engines as you want, and there’s no master-slave relationship. They’re all peers, all coordinating to fire jobs and run workflows. If one goes down, the others continue running and continue cooperating. There’s no sense of, when the master is down, we need to switch over to the failover machine. There’s none of that. The advantage of that, of course, is configuration is simpler, and you get more scalability.