Workload automation projects generally start simple. The first step is often setting up the schedule of when to run jobs. Schedules can start out straightforward, but over time the kinds and types of schedules needed to be addressed evolve and grow increasingly complex.
A commonly used time expression language is that of Unix Cron. We here at Flux have extended the Cron syntax to support the wide variety of schedules seen by our customers. Flux time expressions have 11 fields. Position (1) ms (2) sec (3) min (4 )hr (5) day_in_month (6) month (7) weekday (8) day_in_year (9) week_in_month (10) week_in_year (11) year as well as support for relative time expressions (e.g., +35s, +1m, +2H, +7d, +2w, +1M, +1y)
Consider the following kinds of schedules and their associated time expressions:
- Run a job on a specified date or time, such as midnight on the fifth day of each month, or every 15 minutes between 9 am and 5 pm. [0 0 0 0 5] and [0 0 0,15,30,45 9-17]
- Run a job every 15 hours from now for only Monday – Friday [0 0 0 +15 * * MON-FRI]
- Run a job at two or more times that are unrelated to each other such as midnight and 4:45 am.
[(0 0 0 0|0 0 45 4)] - Run a job on a date or time relative to now (or another date and time), such as 2 months from now or every 10th Wednesday [+2M] and [0 0 0 0 * * WED * +10]
- Run a job starting from a start date and time to an end date and time, firing at regular intervals such as January 1-15 firing at 3 PM. [0 0 0 15 * * * 1-15]
Time expressions support a wide range of scheduling needs, but how does one define holidays, vacations, or other intervals when schedules should, or should not run?
BUSINESS INTERVALS AKA BUSINESS CALENDARS
A Business Interval, generally known as a “Business Calendar,” defines periods of time. These intervals are then applied during the scheduling of workflows to determine whether a workflow may (or may not) run. Unlike ordinary calendars, Business Intervals are not required to be oriented to calendar days and may even have resolution to the second. Picture the calendar as a timeline running from left to right – Business Intervals define the ranges of time that are either included or excluded. Included ranges of time define when workflows may run, and excluded ranges of time define the periods that the workflows may not run.
The “timeline” concept of business intervals means that they do not need to be confined to 24-hour periods – even if it crosses from one calendar day to the next.
Business Intervals can be applied to time expressions to help them calculate their future firing times. Intervals can be used in timer and delay triggers, in calculating timeouts for actions or triggers, and to specify an allowable time window for a File Exist Trigger to monitor for files.
DEFINING BUSINESS INTERVALS / BUSINESS CALENDARS Business Intervals are themselves described with time expressions. They are generally a collection of dates and times using the time expressions that define the allowed processing intervals. For example, here are the time expressions for U.S. Federal holidays. How these work is described further below using shifting.
NEW_YEAR = "^y?sat{<fri}{}?sun{>mon}{}";
MLK = "^y@3mon";
PRESIDENTS = "^y>feb@3mon";
MEMORIAL = "^y>may$M<mon^d";
INDEPENDENCE = "^y>jul+3d?sat{<fri}{}?sun{>mon}{}";
LABOR = "^y>sep@mon";
COLUMBUS = "^y>oct@2mon";
VETERANS = "^y>nov+10d?sat{<fri}{}?sun{>mon}{}";
THANKSGIVING = "^y>nov@4thu";
CHRISTMAS = "^y>dec+24d?sat{<fri}{}?sun{>mon}{}";
USING BUSINESS INTERVALS / CALENDARS
Calendars add meaning to dates. A day can now, for example, be a business- or non-business day, a weekday, a weekend, or a holiday. A time expression of “0 0 0 0 5” (the fifth day of the month) can be amended to “0 0 0 0 5b” to now point to the fifth business day of the month. The “b” indicates the value is to be calculated with the associated business interval.
SHIFTING
NEW_YEAR = ^y?sat{<fri}{}?sun{>mon}{}
Take when New Year’s Day is celebrated as a holiday in a business calendar If it occurs on a Saturday, take off the preceding Friday. If it occurs on a Sunday, take off the following Monday. This is referred to as date “shifting.”
For instance, to run a task on the first business day on or after the 10th calendar day of the month, use “10>1b” in the Days-of-month column. If the 10th calendar day of the month falls on a holiday, then the “>1b” portion will “push” the expression ahead to the next business day. On the other hand, if the 10th calendar day of the month is already a business day, then the “>1b” portion does nothing, because the 10th calendar day of the month is already a business day.
These techniques and symbols can be used in any of the Cron-style time expression fields, not just Days-of-month and Day-of-year fields.
ROLLOVER
FIRST_MONDAY_ON_OR_AFTER_THE_28th = 0 0 0 0 28»1MO
The rollover operator (“»”) provides similar functionality to the right shift operator (“>”). The difference between rollover and right shift is that when the rollover operator attempts to satisfy the constraints of the Cron-style time expression, it can “rollover” into the next Cron column on the right in search of a match, unlike the right shift operator.
SUMMARY
The combination of powerful time expressions and business calendars yield a flexible scheduling platform that address complex scheduling needs. Embedded within Flux is a time expression tester that makes defining and testing time expressions easy.
