The Workflow of Workflows
The management of workflows at an enterprise level
Workflow technology is both immense and far-reaching. There exists a deep and diverse set of software applications, books, conferences, and standards committees related to workflow technology. Workflow technology is not simply a specific tool, a software platform, or a specific acronym (e.g., CPM, BPM, BPEL, etc.) The promise of workflow technology exists in providing enterprises with more flexible, controlled, and consistent services. For the software engineer or IT Operations Manager, adopting workflow technology involves selecting and reviewing workflow engines, workflow design methodologies, management of workflows, and workflow standards and graphical modeling techniques, among others.
Independent of the design, testing, and execution of workflows, effectively extracting value from workflow technology requires active management. Smart technologists realize that workflow is more than a pretty diagram and some running workflows.
Workflows need to be managed.
When an enterprise adopts workflow technology, workflows quickly develop and proliferate. Workflow technology provides a new tool in the IT toolkit, and as with any new tool, opportunities are sought to utilize the new tool, sometimes without regard to appropriateness, scale, or manageability. It is not uncommon for an enterprise to develop hundreds of workflow templates and deploy many thousands of executing instances of these workflows. Keeping control over such proliferation requires one explicitly consider how to manage these workflows. Such management requires one to think of their enterprise’s ‘workflow of workflows.’
This workflow of workflows, in its simplest form, consists of the following workflow related tasks. In many respects the development of a workflow is similar in structure and activity to the construction of any piece of software. Workflow development requires:
- Process review
- Workflow analysis
- Workflow modeling
- Workflow Design
- Workflow implementation, testing, and deployment
- Workflow modification, maintenance, retirement, and enhancement
- Workflow submission, execution, and monitoring
- Workflow promotion and demotion across environments
- Workflow testing
- Workflow debugging and tuning
Numerous workflow design processes exist to facilitate the creation of workflows. These processes are generally well documented and supported with graphical models, varying degrees of formality, and associated tools. For example, one can select from flowcharts to Petrinets to BMPN models.
While creating individual workflows can be straightforward, many find out quickly that building a library of reusable and flexible workflows is not so straightforward. For example, the first version of a workflow generally makes assumptions about how the workflow will be used and by whom. Often the workflow designer is not a ‘workflow expert’ and will make mistakes or limit future flexibility in their workflow design. They may not properly consider parallel activities within the workflow, handling errors across nodes in a distributed workflow cluster, or properly address workflow failover and recovery. One oft-encountered mistake includes hard-coding values into workflows that should be made configurable at run time.
Such trial and error workflow design quickly exposes the need for workflows to be modified and enhanced often, sometimes with quick promotion into production. Suddenly workflow management issues of version control, software testing, workflow promotion and demotion (from developer to testers to production usage) become highly visible and pressing concerns.
Naming workflows is another important workflow management issue. Often workflow names are chosen by developers. If two developers in two different departments define a review workflow, a naming conflict occurs. Creating a workflow naming standard, and then categorizing workflows into hierarchies or ‘namespaces’ becomes a management activity, and one that is not well-addressed in workflow design textbooks. Naming workflows also plays a part when submitting workflows to execution. The name of the workflow while executing may include specifics about the customer, or report, or some other attributes that requires the workflow template name to be different than the workflow’s name after it’s submitted for execution.
And finally, managing the execution of hundreds if not thousands of workflow instances requires careful attention to setting up alerts, monitoring workflow events, knowing the execution characteristics of workflows (e.g., their average run time) and ensuring that the workflow environment is defined and configured in a manner that debugging and resolving issues is straightforward and effective.
Finding the Right Fit for Managing Your ‘Workflow of Workflows’
What factors should you consider in taking a more holistic view of managing your enterprise’s ‘workflow of workflows.’
- Where are the Workflows Stored and Maintained? A central, database-managed store of workflows simplifies and extends the degree of control and management that can be performed. Keeping workflows in text files is adequate where the number of workflows is small, but as more and more workflows are developed the management overhead becomes untenable. A central store of workflows provides greater oversight and reuse, while providing better control over workflow promotion and testing.
- Is There a Convention to Help Manage Workflow Names? Consider the following as one possible approach, although support by your selected workflow technology may provide alternate approaches. A naming convention using ‘namespaces’ allows hierarchical management of workflow names. Just like a file system on a server, namespaces begin at a root level ("/"), followed by a series of 0 or more "folders" or "branches", then finally a unique name for the workflow itself, to identify the workflow among other workflows on the same branch.
Namespaces can not only used to organize workflows according to their branches, but also control how a particular workflow will relate to other concepts like the runtime configuration settings or business calendars. For this reason, consider storing workflows that share many common properties or similar functionality under the same namespace to allow easy configuration.
- What is the Mechanism to Promote and Deploy Workflows? Ideally, workflows, once defined and tested, should move up the chain from development to QA to production without suffering copying errors. The actual workflows and configurations should be exportable and capable to be placed into a version control system for audit and configuration management.
Also consider how workflow templates are transferred from development to QA to production. Is there a straightforward and integrated promotion mechanism, or is the effort highly labor-intense and manual?
- How does one Manage the Execution of Workflows? As noted at the beginning of this article, the people involved in automated workflows are as important as the software that performs those transfers. At various points in the workflow lifecycle, IT operations staff must be engaged to troubleshoot delayed and failed workflows, and even apply business judgments within a workflow to address exceptional situtations – such as a critical shared resource being unavailable or down.
- Workflow Submission: Support for naming the workflow different than the original workflow template name, and support for provisioning the workflow with specific runtime configuration properties or ad hoc information.
- Workflow Control: Support for starting, pausing, restarting, recovering, canceling, expediting, and prioritizing workflows is essential.
- Logs, Reports, and Workflow Run History: Being able to research events and overall workflow statistics provides for timely issue resolution and allows for configuring automated error handling and recovery workflows.
Effectively exploiting workflow technology is much more than simply procuring a workflow platform and then creating workflows. The design and development of a workflow is a small factor in the overall lifecycle of a workflow. Actively managing all the elements in your ‘workflow of workflows’ will allow you to maximize the benefits associated with your workflow investment.
The Flux software platform orchestrates file transfers and batch processing workflows for banking and finance. First released in 2000, Flux has grown into a financial platform that the largest US, UK, and Canadian banks and financial services organizations rely on daily for their mission critical financial systems. Flux provides Electronic Bank Account Management (eBAM) solutions for banks. Electronic bank account management replaces slow paper-based processes with electronic efficiencies, reducing human errors and providing greater transparency into bank and corporate operations.
Banks that offer an eBAM solution possess a critical market advantage in their efforts to expand and retain their corporate customer base.