Orchestrating the Processing of Digital Media
A common workflow for orchestrating digital media files
The nature of digital media varies widely, from check images to digital photos to PDF- formatted contracts and legal documents. Yet the activities performed against digital media are often common and repeated.
These repeatable activities are best expressed as a workflow — and often as a set of workflows.
Drawing on more than ten years’ experience of working with a wide variety of customers who create workflows that do a great many different things, Flux found that the workflows that involve digital media almost always contain a common set of activities.
Workflows involving digital media often involve orchestrating some set of the following activities. By reviewing these common workflow steps, listed below, you can save time and frustration by following these best practice workflow steps.
Common Workflow Steps
Register: registration or arrival event that logs receipt of the digital media file into a database.
Decrypt / Decompress: Optionally, decryption or decompression is performed depending upon the nature of the file.
Validate Against Standards: A general validation of the digital media and its metadata is conducted. Both components must be tested to ensure they can be parsed, and that both conform to agreed-upon standards, often defined by published standards. These standards define elements such as required fields, range checks, and media size checks.
Duplicate Check: Duplicate checking is done to ensure that the metadata or digital media has not been received previously.
Acknowledge Receipt: In many processes an acknowledgement is sent indicating the file has been received successfully.
Validate for Use: An enterprise specific validation of the metadata and media is executed using the enterprise’s specific business rules and validations, above and beyond those called for in the general ‘Validate Against Standards’ step above.
The general validation assures the file conforms to the external standard, but the enterprise may have additional constraints (e.g., required fields, list restrictions) that must be applied against the metadata or media.
Process: Other processes may also be executed against the metadata and media to determine its completeness or fitness for use. In digital content management systems, processes may execute that validate media format and content (e.g., TIFF tag validation, valid PDF form content, ensuring images are not too light or too dark, or rotating images to correct their orientation).
Review / Moderate: In instances where either the metadata or the media is found to not meet enterprise standards, the metadata or media is submitted for some form of review (either automated or manual). For some media types this review operation may be referred to as ‘moderating’, as in moderating for inappropriate content.
Depending upon the specific application, files that fail this step may be submitted to a repair or editing process where the media and metadata can be corrected or otherwise edited to allow it to pass this review.
Reject: At some point in the process files may need to be rejected back to the sender. Such rejection generally involves the creation of a rejection notice with a set of rejection reasons provided to the sender so that they can resolve the issues and resend the file.
Publish: Content from the metadata and media often has to be combined in some manner to feed other systems, such as web sites or backend billing and posting systems.
Archive: In many instances the metadata and media are indexed and stored in long-lived data retention and storage systems, in some cases up to many years.
Implementing the Workflow is Neither Trivial Nor Simple
While the workflow described above is consistent and repeatable, implementing the workflow is neither trivial nor simple. Beyond just the execution of the workflow one must address:
- Monitoring potentially thousands of these processing concurrently
- Restart and recovery concerns
- The need in many instances to perform ad hoc actions
- Scaling, load-balancing, clustering, and failover are also key attributes to large-scale implementations of this particular workflow.
- Service level agreement (SLA) monitoring and notifications (e.g., via email, text, or integration with enterprise service management tools) of exceptional events and SLA violations also cannot be ignored.
Flux’s file orchestration capabilities make it ideally suited for high-volume, mission-critical applications involving digital media. One such application is a check image exchange system that Flux underpins, handling the exchange of millions of checks daily between large U.S. financial institutions. Check image exchange involves the exchange of check images and associated metadata in an industry standard file format. These files are exchanged across a variety of networks between banks, on ATM networks, and between corporations and their banks. The above mentioned processes are all in play in the execution of this workflow against check images.
In this check image exchange system, the names of the processes may be different in the workflow, but the fundamental behavior of each process is still present. For instance, in check processing the ‘Validate for Use’ step is referred to as ‘Sort Pattern Edit.’ ‘Review and Moderate’ is referred to as ‘Reject / Repair.’
File Orchestration is a Form of Workflow
File orchestration exists as one of the many forms of workflow occurring in enterprises today. One size does not fit all workflows. Selecting the appropriate platform for this key and mission-critical workflow is essential in addressing the wide range of digital content processing occurring in today’s enterprises.