File Orchestration Platforms
Moving Beyond File Processing to File Orchestration
File processing – all too often equated with COBOL batch-processing – is a key and often overlooked facility of modern financial systems. Although it may be considered 'low-tech' in this age of streaming video, tablet and mobile phone apps, big data, and high-speed messaging, much of the financial processing taking place today is processed via files.
These files may be log files being submitted for review by analytics engines, or a collection of bank checks being submitted for payment, or a set of financial transactions being returned as a report to a law enforcement agency triggered by a search warrant.
Nonetheless, in some manner, shape, or form, a vast amount of processing is conducted with files.
Today, file processing is generally addressed in a fragmented, siloed, application-byapplication basis. Each application has an isolated view of the file itself — and what is needed for the application’s processing. Each application has its own private knowledge and understanding of the file.
The notion of some central facility directing the processing of files is seldom explicit — such control is often buried in task schedulers, file transfer scripts, and sometimes even the files names themselves (i.e., changing file names is indicative of processes being executed and completed).
As the complexity of modern systems continues to increase, the need for centralized control and orchestration of an enterprise's files grows ever more pressing. File processing, while almost as old as data processing itself, has yet to achieve this level of control.
To describe such control in the web services and business process domains, the serviceoriented architecture and business process modeling communities both developed and refined the term 'orchestration'. Refining the 'orchestration' definition from those communities to something specific for file processing we have:
- File Orchestration is the ability to control operational flows and activities based on business rules, especially in multi-application systems.
- File Orchestration includes file transfer and the ability to apply automation to file processing such as triggers, schedules, explicit calls, and chained calls to solve a business problem.
- File Orchestration involves complex workflows, service level agreements, transformations, messaging, alerts, and other attributes of complex enterprise systems.
- Finally, File Orchestrations themselves can be described and assembled from a set of reusable and descriptive patterns describing their behavior and capabilities.
From File Orchestration to File Orchestration Platforms
To paraphrase SOA Patterns — Orchestration:
"An orchestration platform is dedicated to the effective maintenance and execution of business process logic. Modern-day orchestration environments are expected to support sophisticated and complex service composition logic that can result in long-running runtime activities."
Considering the importance of file processing in today's information systems, one could justify file orchestration with the same rationale as orchestrating business processes or web services. As an adjunct to such a platform – and to refine the above definition:
A file orchestration platform is dedicated to the effective maintenance and execution of file processing logic.
File Orchestration Platform Capabilities
Searching for 'file orchestration patterns' on Google brings up a dearth of results. But many seasoned technicians can quickly recite a list of commonly performed file processes. These include, for example and in no particular order:
|File Splitting||File Extraction||File Replication|
|File Conversion / Transcoding||File Reformatting /
|File Matching and Merging|
|File Concatenation||File Aggregation||File Validation|
|File Compare||File Parsing||File Summarization|
|File Archiving||File Sorting||File Loading (to databases and
|File Generation/Creation||File Ingestion||File Injection|
|File Copy||File Processing - local/remote||File Deletion|
|File Renaming||File Encrypting and Decrypting||File Versioning|
|File Scheduling||File Detection||File Transfer|
Each file process provides value to multiple enterprise business processes. The ‘File Compare’ process, for example, provides value in business processes involving change management, audit and compliance, and internal controls.
Having a catalog of file processes from a business process perspective is valuable. Such a catalog would contain for each file process, for example:
- A definition of the process
- The impact of the process, and its applicability, to business and technology usage
- Usage statistics of where the process is used
- A list of example usages within existing and future applications
- A definition of metrics and key performance indicators (KPIs) associated with the process
Reviewing and refining this file process catalog leads to insight into each process and how processes can be combined into file orchestrations and business workflows. With careful attention paid to the issues regarding high volume and/or mission-critical processing, these file orchestrations lead to significantly simpler, robust, and reliable workflows.
Sample file process catalog entries include:
Definition: For each file located at a source file location, take a delimiter — such as a length, or a character, or a record ID, or a regular expression, or ... — and split the file into multiple files, named using an output formatter.
Utility: Useful in instances where the contents of a file need to be split out to support multiple applications, or where the contents of the file can be split into multiple files for processing in parallel, or for processing portions of a file in an expedited manner.
Impacts: This file process impacts those business processes that are fed payment streams that contain items that can be cleared in multiple manners, such as via wire, ACH, or conventional check presentment. X9.37 file check image exchange is one such business process.
Usages: Example usages within existing and future applications include exchanges involving peer-to-peer exchange via the SVPCO DTA and Endpoint Exchange, the Federal Reserve, and correspondent bank image exchanges.
Metrics: Key process indicators and metrics include:
- The number of and percentage of incoming files where the file split process is applicable
- The associated value with performing the split (reduced clearing costs, offering of new customer products based on early availability of data to review processes, …)
The costs and risks associated with performing the split (e.g., reconciliation and tracking of multiple files instead of one, development costs of the split, day-to-day operational costs of executing the split).
Definition: For each file located at a source file location, apply a converter against the file and output another file, named using an output name formatter.
Utility: Useful in instances where the contents of a file need to be converted into another format for delivery to an application or to a third-party.
Impacts: This file process impacts those business processes that require data format conversion or content conversion. This happens frequently when proprietary file formats and industry-standard file formats have to be exchanged or otherwise processed. There exists a significant customer impact in supporting this process as it allows customers to provide files in their format of choice – without requiring them to convert their files or install new software.
Applicability: In check image exchange, this process can be used to convert a file that conforms to one industry standard into another industry standard, such as converting CPCS strings into X9.37 files. For corporate treasury applications this process can be utilized to eliminate or reduce the amount of development work a corporation must commit to in order to interface with the bank’s existing corporate treasury application suite.
Usages: Example usages within treasury management applications for new customer onboarding include taking existing proprietary formats that the corporation was previously creating for their prior vendor, and converting these formats into a format compatible for bank processing.
Definition: For each file located at a source file location, use an archiver to archive each file when a supplied trigger fires.
Utility: Required where files need to be added to long term archives, such as a bank’s alltransactions file, or in instances where files need to be archived for potential reprocessing at a later time, such as end-of-month.
Impacts: This file process impacts those business processes that require data storage for archival purposes.
Applicability: This happens, for instance, with check image archives where the check image must be retained for some number of years within a digital archive, maintained by either the bank or a third-party provider such as Viewpointe.
Usages: Example usages include delivery of images to an internal or third party image or document archive, as well as the submission of payments-related transaction data to a bank’s All Transactions File.
The Business Value of a File Orchestration Platform
The business value of a file orchestration platform lies in the facilities it provides to standardize and improve the reliability of fundamental file processing activities. Financial institutions constantly grapple with disparate industry-standard and proprietary file formats, and varied channels to receive and deliver files. Challenges surface in managing the receipt, processing, converting, and delivery of this diversity. Providing a consistent means of addressing this diversity is a key tenet of a file orchestration platform.
In a manner analogous to a web application server, a file orchestration platform wraps each of the above processes with retry and error handling capabilities, a storage mechanism to maintain each of the above patterns and make them reusable in multiple circumstances, load balancing and distributed processing, monitoring, and service level tracking and alerting.
A file orchestration platform must also address the ‘no one size fits all’ paradox. An approach that works well for processing small text files may not fit for processing very large (gigabytes in size) binary files containing images or digital media. Any effective file orchestration platform must provide a robust tool set of capabilities to address the diversity of processing it may encounter. Key attributes of a file orchestration platform consist of, for example:
- Load balancing of work
- Clustering across multiple physical and virtual processors
- Fine-grained partitioning and allocation of work across a cluster — allowing work to be processed on machines most appropriate for the file processing required
- Parallel execution of processes
- The facility to delegate tasks to other resources
- Built-in and user-defined facilities for file transfer, file processing, and file validation
- File-by-file process monitoring
- Automated retry and recovery of file processing exceptions
- Centralized storage and management of file workflows
- Secure processing of files and fine-grained access controls to the workflows and processing engines that govern file processing
Taken in their entirety, these facilities comprise the business value available from a file orchestration platform that is not otherwise achieved in existing application servers or homegrown solutions.
A Sample File Orchestration
Consider a common file orchestration - that of routing a payload of information (such as a payments file or a report) to a destination based on routing information maintained in an enterprise database. Such file processes are common in numerous systems, such as the delivery of disclosure forms for financial transactions and the delivery of reports originating from another bank system. The workflow is kept intentionally simple. In a real environment, for example, the payload file would usually need to be encrypted before delivery.
Flux — A File Orchestration Platform
Flux assists enterprises in provisioning, onboarding, scheduling, tracking, and reporting an enterprise’s file orchestration processes. These orchestrations vary from simple file transfers to highly complex workflows involving extensive processing, many routes, varied alerts, and complex decisioning. Flux has deep expertise in facilitating financial institutions (e.g., banks, lenders) and financial service providers (e.g., software vendors and service bureaus) in effectively orchestrating files to create new revenue opportunities and facilitate expense reductions.
Flux’s file orchestration facilities have evolved over the past 12 years, providing numerous institutions significant competitive advantage in delivering highly available and reliable solutions to their customers.
The cornerstone of Flux’s technology is its orchestration engine. Optimized for file orchestration. Flux orchestrates the creation and delivery of an enterprise’s content to its customers, partners, and internal departments. Flux’s file orchestration facilities simplify and extend financial institutions’ operations by providing:
- Graphical, browser-based, drag and drop workflow designer
- Clustering support for high volume or tight SLA processing loads
- Load balancing and failover for scalability and high availability
- Connectivity options for multiple delivery channels using FTP, SFTP, and other protocols
- Support for defining service level agreements
- Written in pure Java and portable to any platform with an available JVM
- Systems management functionality to provide proactive monitoring
- Seamless interfaces with a variety of host legacy systems, enterprise systems, networks, messaging types, relational databases, and Web services
- High volume, 24x7 continuous processing support for mission-critical business needs throughout the enterprise
File orchestration is the key next step in supporting highly reliable and repeatable business processes involving files. A file orchestration platform standardizes and provides a repository for a catalog of common file processes that are then designed into executable workflows supporting robust file orchestrations. The Flux file orchestration platform provides the key features required of file orchestration platforms in general, as well as providing many advanced capabilities to better service banks and financial service providers in satisfying their unique needs.
The Flux software platform orchestrates file transfers and batch processing workflows for banking and finance. First released in 2000, Flux has grown into a financial platform that the largest US, UK, and Canadian banks and financial services organizations rely on daily for their mission critical financial systems. Flux provides Electronic Bank Account Management (eBAM) solutions for banks. Electronic bank account management replaces slow paper-based processes with electronic efficiencies, reducing human errors and providing greater transparency into bank and corporate operations.
Banks that offer an eBAM solution possess a critical market advantage in their efforts to expand and retain their corporate customer base.