The Enterprise Scheduler
The IT Manager’s Quest for Sleep
This is not a story of vagaries and euphemisms. Unfortunately, this story is also often-told. This is a story starting with anguish and suffering. It may end with a happy ending, it may not. But it’s all about you – The IT Manager.
Once Upon a Time
You know the story. For all we know, you may be in the midst of this story. As a development manager or IT manager, you get the late night call. Job “XYZ” is running too long, and impacting multiple jobs. In a bank the dreaded words are “we cannot post.” Other enterprises have different phrases or different events (e.g., “we cannot close the month”) but in each case these words elicit a sense of dread. You’re not going to get any sleep, you’re going to have to rouse a legion of people from a sound sleep, and you’re looking forward to a litany of “root cause” meetings and incident reports in the coming days. What happened? You just potentially experienced:
- Unexpected downtime, possibly caused by human error (i.e., one of your staff)
- Your operations staff complaining they have no control over production processing, making recovery difficult
- You just got a Monday morning “surprise” due to weekend errors
- No one was notified about delays and errors in a mission-critical job stream
Off with their Head!
And now, for whatever reason, it’s deemed your fault! Even if it’s stated in more euphemistic terms, This is not a story of vagaries and euphemisms. Unfortunately, this story is also often-told. This is a story starting with anguish and suffering. It may end with a happy ending, it may not. But it’s all about you – The IT Manager.
If the event is severe enough (one customer refers to their event as “The Saint Valentine’s Day Massacre”, a task force or “consultant” is summoned in search of a solution. They are tasked with, for lack of other wording, to find you a better way to do your job. They do their research and conclude, from many sources, that job automation (or the keyword du jour e.g., IT process automation, workloadautomation, even self-healing systems) has shown significant benefit in enterprise activities such as:
- Backup and restoration
- Disaster recovery/failover
- Service request/fulfillment
- Incident management/event response
- Data movement
In addition, they cite a litany of opportunities for savings. Their research shows opportunities for:
- Savings due to your potentially using fewer or less costly personnel
- Savings realized by your reducing or eliminating manual, unnecessary or repeating processes
- Savings enjoyed by your no longer missing SLAs, and reduced customer dissatisfaction
- Savings on your meeting regulatory and other compliance issues
- Savings through your purchasing fewer or smaller quantities of software and licenses
- Savings on annual software maintenance costs
- Savings on homegrown software development, testing, and maintenance
- Savings through your use of fewer hardware, virtual, database or OS platforms
Here Ye, Here Ye
The business requirements get cited. A proclamation is made. You are directed to find a way to:
- Simplify the way you create and control batch and on-line processes
- Align views of IT processes with the business
- Improve your customer response times and IT resource utilization
- Reduce your resolution times
- Get more out of your existing assets
- Schedule jobs to execute in a consistent and repeatable manner
- Reduce the number of manual intervention and errors
- Lower and standardize your IT staff maintenance tasks
- Increase your responsiveness to business demands
- Better utilize available systems and the network
- Decrease risk
- Rapidly update jobs
And so the edict is issued. Senior management demands that you procure a job scheduler. But not a simple job scheduler, an “enterprise” job scheduler! You convene a team to interview internal and external customers of the enterprise on a list of requirements. Requirements like – well like the following:
|BENEFIT TO THE ENTERPRISE||ACHIEVED HOW?|
|Simplify the way you create and control batch and on-line processes||Through a graphical user interface and easy-to-use scripting|
|Align views of IT processes with the business||By correlating IT job names to business-relevant names and categories|
|Improve your customer response times and IT resource utilization||With real-time, event-driven automation|
|Reduce your resolution times||With proactive intervention and remediation of problems|
|Get more out of your existing assets||Through seamless integration of applications and systems|
|Schedule jobs in a consistent and repeatable manner||Across multiple operating systems, packaged and custom apps|
|Reduce the number of manual intervention and errors||Via automated response to exceptional events|
|Lower and standardize your IT staff maintenance tasks||Allowing your staff to work on other, more important projects|
|Increase your responsiveness to business demands||By synchronizing job executions with external application or system events|
|Enhance systems and network performance||To better meet service level goals|
|Decrease risk||Through improved security and resiliency|
|Rapidly update jobs||By allowing changes to be done quickly|
And the Quest Begins
And your quest for a tool or technology begins. Taking the above requirements, you begin to define the attributes of the technology you are looking for. For instance:
|BENEFIT TO THE
|Simplify the way you create and control batch and on-line process||Through a job management and monitoring user interface and graphical workflow design / easy-to-use scripting||Fat client
Workflow definition without writing code
|Align views of IT processes with the business||By correlating IT job names to business-relevant names and categories||Naming convention
Category or Description
Hierarchical structuring and viewing of processes
|Improve your customer response times and IT resource utilization||With real-time, event driven automation||Automation support for conditional branching, dependencies, looping, parallelism, splits, and joins
Event support for cron expressions, database conditions, message queues, web service requests, workflow conditions, and file existence and state
|Reduce your resolution times||With proactive intervention and remediation of problems||Email notification
Automatic web service calls to problem ticket systems
Job event and subscription notifications Audit trail
Easily understood and user-definable messages
Automated recovery processes
|Get more out of your existing assets||Through seamless integration of applications and systems||Agentless deployment
|Schedule jobs in a consistent and repeatable manner||Across multiple operating systems, packaged and custom apps||Relative and absolute time-based scheduling
Scheduling precision at least to the minute
Multiple business calendar support
Dependency management between workflows, time, and external events (e.g. incoming web service calls)
|Reduce the number of manual interventions and errors||Via automated response to exceptional events||Automated error handling
Default and workflow specific handling
Proactive notification of SLA pending and exceeded events
|Lower and standardize your IT staff maintenance tasks||Allowing your staff to work on other, more important projects||Standardized workflows that provide for dynamic configuration from files, database tables, or web service calls|
|Increase your responsiveness to business demands||By synchronizing job executions with external application or system events||Load balancing
Resource dependency management
|Enhance systems and network performance||To better meet service level goals||Distributed processing across nodes – even within one workflow
Pinning particular jobs to specific nodes when required
Dedicating resources to specific jobs for unique requirements – such as leveraging IT appliances.
|Decrease risk||Through improved security and resiliency||Leveraging distributed resources for increased reliability and scalability
Granular access controls with integrated authentication with Active Directory and LDAP
Automated failover and recovery
Warm and cold recovery modes
|Rapidly update jobs||By allowing changes to be done quickly||Real time schedule changes, support for ad hoc submissions, and dynamic job reconfiguration|
The good news is there exists a variety of products that provide significant coverage for these requirements. You are not going where no one has gone before. An obvious place to start is Wikipedia’s List of Job Schedulers. But be aware, with great power comes often a great (as in expensive) price. So balance your wish list with your “I must absolutely have in order to survive” list.
Many vendors have free evaluation software, as well as demonstration sites to view online. But during the selection process, be prepared for the following due-diligence process:
- Prepare an RFP, or at least a clear list of requirements. Steal from above if you like.
- Determine exactly how you will finally qualify a vendor for inclusion in your short list.
- Get a demonstration of the system from the vendor.
- Look at the training outline for vendor’s product.
- Define and share the Proof of Concept requirements with your vendor.
- Schedule and execute the POC.
- Evaluate the results of the POC.
- Document the lessons learned and best practices.
- Make your selection.
- Distill all lessons and practices into a single rollout and deployment process (or process set).
- Pilot your selection on some high value moderate risk workflows using the defined process.
- Evaluate the lessons learned and best practices discovered.
- Refine the rollout and deployment process.
- Rollout the implementation.
- Return to a good night’s sleep.
Happily Ever After?
Declare a win and take solace that your efforts will indeed make you show the world you can do your job better. Your return to getting a good night’s sleep may not necessarily be long-lived for (let’s be realistic) you are an IT Manager!
The Flux software platform orchestrates file transfers and batch processing workflows for banking and finance. First released in 2000, Flux has grown into a financial platform that the largest US, UK, and Canadian banks and financial services organizations rely on daily for their mission critical financial systems. Flux provides Electronic Bank Account Management (eBAM) solutions for banks. Electronic bank account management replaces slow paper-based processes with electronic efficiencies, reducing human errors and providing greater transparency into bank and corporate operations.
Banks that offer an eBAM solution possess a critical market advantage in their efforts to expand and retain their corporate customer base.