The Enterprise Scheduler

This is not a story of vagaries and euphemisms. Unfortunately, this story is also often-told. This is a story starting with anguish and suffering. It may end with a happy ending, it may not. But it’s all about you – The IT Manager.

Once Upon a Time

You know the story. For all we know, you may be in the midst of this story. As a development manager or IT manager, you get the late night call. Job “XYZ” is running too long, and impacting multiple jobs. In a bank the dreaded words are “we cannot post.” Other enterprises have different phrases or different events (e.g., “we cannot close the month”) but in each case these words elicit a sense of dread. You’re not going to get any sleep, you’re going to have to rouse a legion of people from a sound sleep, and you’re looking forward to a litany of “root cause” meetings and incident reports in the coming days. What happened? You just potentially experienced:

• Unexpected downtime, possibly caused by human error (i.e., one of your staff)
• Your operations staff complaining they have no control over production processing, making recovery difficult
• You just got a Monday morning “surprise” due to weekend errors
• No one was notified about delays and errors in a mission-critical job stream

Off with their Head!

And now, for whatever reason, it’s deemed your fault! Even if it’s stated in more euphemistic terms, This is not a story of vagaries and euphemisms. Unfortunately, this story is also often-told. This is a story starting with anguish and suffering. It may end with a happy ending, it may not. But it’s all about you – The IT Manager.

If the event is severe enough (one customer refers to their event as “The Saint Valentine’s Day Massacre”, a task force or “consultant” is summoned in search of a solution. They are tasked with, for lack of other wording, to find you a better way to do your job. They do their research and conclude, from many sources, that job automation (or the keyword du jour e.g., IT process automation, workloadautomation, even self-healing systems) has shown significant benefit in enterprise activities such as:

• Backup and restoration
• Disaster recovery/failover
• Service request/fulfillment
• Incident management/event response
• Data movement

In addition, they cite a litany of opportunities for savings. Their research shows opportunities for:

• Savings due to your potentially using fewer or less costly personnel
• Savings realized by your reducing or eliminating manual, unnecessary or repeating processes
• Savings enjoyed by your no longer missing SLAs, and reduced customer dissatisfaction
• Savings on your meeting regulatory and other compliance issues
• Savings through your purchasing fewer or smaller quantities of software and licenses
• Savings on annual software maintenance costs
• Savings on homegrown software development, testing, and maintenance
• Savings through your use of fewer hardware, virtual, database or OS platforms

Here Ye, Here Ye

The business requirements get cited. A proclamation is made. You are directed to find a way to:

• Simplify the way you create and control batch and on-line processes
• Align views of IT processes with the business
• Improve your customer response times and IT resource utilization
• Reduce your resolution times
• Get more out of your existing assets
• Schedule jobs to execute in a consistent and repeatable manner
• Reduce the number of manual intervention and errors
• Lower and standardize your IT staff maintenance tasks
• Increase your responsiveness to business demands
• Better utilize available systems and the network
• Decrease risk
• Rapidly update jobs

And so the edict is issued. Senior management demands that you procure a job scheduler. But not a simple job scheduler, an “enterprise” job scheduler! You convene a team to interview internal and external customers of the enterprise on a list of requirements. Requirements like – well like the following:

BENEFIT TO THE ENTERPRISE	ACHIEVED HOW?
Simplify the way you create and control batch and on-line processes	Through a graphical user interface and easy-to-use scripting
Align views of IT processes with the business	By correlating IT job names to business-relevant names and categories
Improve your customer response times and IT resource utilization	With real-time, event-driven automation
Reduce your resolution times	With proactive intervention and remediation of problems
Get more out of your existing assets	Through seamless integration of applications and systems
Schedule jobs in a consistent and repeatable manner	Across multiple operating systems, packaged and custom apps
Reduce the number of manual intervention and errors	Via automated response to exceptional events
Lower and standardize your IT staff maintenance tasks	Allowing your staff to work on other, more important projects
Increase your responsiveness to business demands	By synchronizing job executions with external application or system events
Enhance systems and network performance	To better meet service level goals
Decrease risk	Through improved security and resiliency
Rapidly update jobs	By allowing changes to be done quickly

And the Quest Begins

And your quest for a tool or technology begins. Taking the above requirements, you begin to define the attributes of the technology you are looking for. For instance:

BENEFIT TO THE ENTERPRISE	ACHIEVED HOW?	USING
Simplify the way you create and control batch and on-line process	Through a job management and monitoring user interface and graphical workflow design / easy-to-use scripting	Fat client Browser Client Zero-configuration Graphical API REST Scripting Workflow definition without writing code
Align views of IT processes with the business	By correlating IT job names to business-relevant names and categories	Naming convention Category or Description Namespaces Hierarchical structuring and viewing of processes
Improve your customer response times and IT resource utilization	With real-time, event driven automation	Automation support for conditional branching, dependencies, looping, parallelism, splits, and joins Event support for cron expressions, database conditions, message queues, web service requests, workflow conditions, and file existence and state
Reduce your resolution times	With proactive intervention and remediation of problems	Email notification Automatic web service calls to problem ticket systems Job event and subscription notifications Audit trail Log messages Easily understood and user-definable messages Automated recovery processes
Get more out of your existing assets	Through seamless integration of applications and systems	Agentless deployment Agent deployment Multi-platform support Clustering support
Schedule jobs in a consistent and repeatable manner	Across multiple operating systems, packaged and custom apps	Relative and absolute time-based scheduling Scheduling precision at least to the minute Multiple business calendar support Dependency management between workflows, time, and external events (e.g. incoming web service calls)
Reduce the number of manual interventions and errors	Via automated response to exceptional events	Automated error handling Default and workflow specific handling Proactive notification of SLA pending and exceeded events
Lower and standardize your IT staff maintenance tasks	Allowing your staff to work on other, more important projects	Standardized workflows that provide for dynamic configuration from files, database tables, or web service calls
Increase your responsiveness to business demands	By synchronizing job executions with external application or system events	Load balancing Resource dependency management
Enhance systems and network performance	To better meet service level goals	Distributed processing across nodes – even within one workflow Pinning particular jobs to specific nodes when required Dedicating resources to specific jobs for unique requirements – such as leveraging IT appliances.
Decrease risk	Through improved security and resiliency	Leveraging distributed resources for increased reliability and scalability Granular access controls with integrated authentication with Active Directory and LDAP Automated failover and recovery Warm and cold recovery modes
Rapidly update jobs	By allowing changes to be done quickly	Real time schedule changes, support for ad hoc submissions, and dynamic job reconfiguration

Choose Wisely

The good news is there exists a variety of products that provide significant coverage for these requirements. You are not going where no one has gone before. An obvious place to start is Wikipedia’s List of Job Schedulers. But be aware, with great power comes often a great (as in expensive) price. So balance your wish list with your “I must absolutely have in order to survive” list.

Many vendors have free evaluation software, as well as demonstration sites to view online. But during the selection process, be prepared for the following due-diligence process:

• Prepare an RFP, or at least a clear list of requirements. Steal from above if you like.
• Determine exactly how you will finally qualify a vendor for inclusion in your short list.
• Get a demonstration of the system from the vendor.
• Look at the training outline for vendor’s product.
• Define and share the Proof of Concept requirements with your vendor.
• Schedule and execute the POC.
• Evaluate the results of the POC.
• Document the lessons learned and best practices.
• Make your selection.
• Distill all lessons and practices into a single rollout and deployment process (or process set).
• Pilot your selection on some high value moderate risk workflows using the defined process.
• Evaluate the lessons learned and best practices discovered.
• Refine the rollout and deployment process.
• Rollout the implementation.
• Return to a good night’s sleep.

Happily Ever After?

Declare a win and take solace that your efforts will indeed make you show the world you can do your job better. Your return to getting a good night’s sleep may not necessarily be long-lived for (let’s be realistic) you are an IT Manager!