Workload automation sounds simple at its surface. A business consumer (e.g., a department head or a senior executive) may state requirements in “if this, then that” form:
-
If a department budget file arrives, validate it and send to the accounting system for processing.
-
At midnight, backup and index the payroll database.
-
On demand, generate an email to a prospect list of with an attached new product brochure.
They ask information technology staff “What’s so hard about that?” Business consumers are familiar with mobile apps that help them simply script their daily interactions email, calendars, and spreadsheets, and they expect this simplicity to be present in their enterprise applications. In fact, many IT staffs start down this path of writing scripts using the “if this, then that” form, assigning developers or support staff to create command scripts and Windows Task Scheduler and crontab entries to perform routine tasks. But the simplicity of these scripts erode over time. Corporate policies often force such scripts to address additional demands such as:
-
Failover and backup of the scripts and the underlying jobs being executed
-
Security requirements to restrict access to jobs
-
Audit requirements of job creation, submission, pausing, interrupts, and restarts
-
Centralized monitoring of hundreds to thousands of these tasks
-
Error recovery that becomes a mission critical component of the daily operation of the enterprise And as these requirements are addressed the scripts become large, complex, often undocumented, and difficult to maintain. Such scripts are often created by a single person, creating a single point of failure when that individual takes leave or exits the company.
Yet constructing workload automation tasks that are simple enough for the business consumer yet comprehensively support the enterprise’s needs is indeed a reasonable goal. How does one achieve such a goal? Many engineering activities use models to make things simpler to grasp while hiding unnecessary complexities. Models can be graphical, as well as textual and even mathematical.
To better understand how models are useful in workload automation, let’s take the first example requirement above and start to expand what fully supporting it requires:
-
If a department budget file arrives, validate it and send to the accounting system for processing.
-
Listen on the SFTP server located in the company’s data center DMZ for the arrival of a file with the extension .BUDGET.
-
Determine if the file is empty (has zero bytes)
-
If true, send an email notification to the customer that sent the file.
-
Locate the customer ID in the file’s name using a regular expression
-
Looking up email address information in a customer database using the customer ID as a key.
-
Update the customer database record with a note that an empty file has been sent, and cannot be processed until a corrected file is sent.
-
Append to the file the extension ‘PENDING CUSTOMER RESPONSE’. …
Continuing to expand the requirement can lead to many pages of details – burying the consumer in the implementation details of – what is perceived by them – a simple and often performed need. Not shown in the above text are all the paths that may be taken in the event of errors or exceptional condition, e.g., the SFTP server is down, or no file is present at the expected time. So what’s needed to manage this complexity back to the supposed simple request of the business consumer? Two powerful strategies for managing complexity are those of abstraction and information hiding. From Techopedia,
-
Abstraction: “the act of representing essential features without including the background details or explanations.”
-
Information-Hiding: “a way in which clients could be shielded from internal program workings.”
Abstraction and information-hiding work in concert to provide flexibility and simplicity by focusing attention on the essential details of the work being automated while hiding the lower-level details of how that is accomplished. Powerful yet simple to use workload automation platforms provide modeling features that hide low level details and focus on the key items of interest (the key “abstractions”) in a request. For example, workload automation models should consider the following common workflow concepts:
-
Triggers – an abstraction that waits or listens for an event to occur, and when that event occurs they “fire” or execute, returning a result appropriate for the trigger. A timer trigger, for example, listens for the arrival of a time and when that happens it fires and flows to the next step. The arrival of a file (which returns information about the file) and the existence of the database in a given state (which returns true or false depending upon if the database condition is met) are other instances of triggers.
-
Actions – a different abstraction that performs a service, such as running an operating system command or performing a database query. Actions return results too, such as the exit code of running a command, or a list of rows retrieved from a database query.
-
Flows – flows are the connectors, or “glue”, between actions and triggers. Flows contain the results generated by actions and triggers (and data you may want to add) and routes this information to the next steps.
-
Workflows – workflows are flowcharts consisting of the above items. In the context of a workflow the above actions, triggers, and flows interact to generate an intended result.
Each of the above abstractions hides, and makes visible when needed, its own implementation details. Triggers may hide/expose the conditions when they fire, or the interval that they wait before firing. Actions may hide/expose the command they are to run, or the maximum time the command should be allowed to run before timing out.
Workflow designers through their models can exploit abstraction and information-hiding to facilitate the creation of understandable and easy to use workflows without overwhelming the business consumer with low level details. For example, the example requirement can be modeled with the following, either graphically or in text:
-
Define a trigger that checks for a file’s presence. Within that trigger specify the details of file server host names, file transfer protocol, user credentials, and directories and file names to scan for.
-
Define an action that performs a validation routine against the file. Within that action specify the command or program code to use and additional details as to working directories, timeout conditions, where to route error or output messages, etc.
-
Define an action that copies the file to a location the accounting system recognizes. Within that action specify server name and directory names to move files to, as well as renaming the file with a suffix or date processed extension.
-
Define flows between the above triggers and actions. Then define the properties of the flows, such as timeout conditions, which flow to take depending upon the result of the prior action or trigger, and exceptional or error flow conditions.
The above models can be constructed using a number of modeling approaches such as text (e.g., business rules or formal executable specifications) or graphics. For example, a graphical modeling tool that supports creating icons (for the abstractions) and assigning properties to them (for the implementation details) is one approach. Visio and draw.io are two tools one could consider. But workload automation models created using these tools would not actually run workloads. Ideally a workload automation “platform” would provide support for such modeling, as well as provide an environment within which these models would execute. Below is one example of modeling just the abstractions, while hiding the low-level details, in a common workflow.
In Summary
Describing requirements using “if this, then that” is a simple, but not sufficient, means of modeling workload automation. A more powerful and comprehensive approach can be achieved by isolating the fundamental concepts of workloads (abstractions such as triggers, actions and flows), and modeling these into workflows. Ideally a workload automation “platform” supports directly executing these models making the distance from concept to running workflows a mouse click away.
