What to Run Next: Workflow Order of Execution within Flux
Flux provides a powerful set of facilities to control the order of execution of workflows on a Flux engine. When multiple workflows are waiting to be executed on a Flux engine, the engine must determine which workflow should be executed next. To make this decision, the engine considers three primary factors: Eligible Execution Time, Priority, and First-In-First-Out Scheduling.
Eligible Execution Time: the time at which a workflow declares that it is ready to execute. Most actions are ready to execute immediately once workflow control reaches them. Most triggers, on the other hand, wait until a certain time before they are ready to execute, at which time they check to see if a condition particular to that trigger is satisfied. Note that a workflow may not actually fire at its Eligible Execution Time due to engine availability or scheduling conflicts. Moreover, if a workflow is paused, it will not execute, regardless of its Eligible Execution Time.
Priority: If two workflows are both ready to execute — that is, the wall clock (the current time) is at or beyond their Eligible Execution Times — the workflow with the higher priority will always run first. Workflow priorities are useful when not all of your workflows can run at once (for example, if processing power is limited) and you need to ensure that certain workflows run before others. Since not all of your workflows can run at once, you will often to need to ensure that the most important tasks are completed first. Workflow priorities can be set directly on a workflow using the Flux Designer or the Flux Java API, or through the runtime configuration of the Flux engine. The lower the number, the higher the priority; a priority of 10 takes precedence over a priority of 100. For more information, see Runtime Configuration and Runtime Variables.
If a workflow has a priority explicitly set using the Designer or API, that priority will override any priority set in the runtime configuration. If a null workflow priority is used, it will allow the runtime configuration priority to take precedence. Workflow priorities can be stored in an engine’s runtime configuration. Each branch in the runtime configuration tree can contain a PRIORITY property, which specifies the priorities of all workflows in that branch of the tree. If a PRIORITY property is not explicitly set, the engine searches higher branches in the runtime configuration until it finds a PRIORITY property. If no explicit PRIORITY property is found, even at the root of the tree, then the workflow defaults to a priority of 10.
First-In-First-Out (FIFO) Scheduling: If enabled, FIFO Scheduling controls whether workflows are scheduled for execution in a first-in-first-out manner. When a workflow is originally submitted to an engine, the current timestamp is recorded as the workflow creation timestamp. Next, when a workflow’s Eligible Execution Time permits it to execute, all such eligible workflows begin execution in the order of their creation timestamps. Priorities take precedence over FIFO scheduling; if an older and a newer workflow are eligible to execute and the newer workflow has a higher priority, the newer workflow begins executing first.
Other factors may also affect the order of execution. For instance, It is possible that higher priority workflows will run so frequently as to prevent lower priority workflows from running at an acceptable rate. This behavior is called starvation, that is, lower priority workflows can be starved or prevented from running as frequently as you would like. By default, Flux permits starvation, because it often does not cause any serious problems, and it is sometimes desirable.
If starvation impacts you adversely, you can enable the Flux configuration property FAIRNESS_TIME_WINDOW. This property contains a time expression that indicates how frequently starved workflows should have their priorities increased by 1 (the effective priority). (To increase a workflow’s effective priority, its numerical value is decremented by 1.) Eventually, such workflows will reach an effective priority of 1 and be eligible for execution before all other workflows. After firing, such workflows revert to their original priorities and are again eligible to have their priorities increased if they starve. This anti-starvation behavior is called fairness.
Note that whenever a trigger fires internally, even if it does not fire out of the trigger and force execution of the workflow to continue to the next action, that workflow’s effective priority is reset to its original priority.
Also note that because starved workflows have their priority increased by one (their priority value decremented by one), appropriate priorities should be given at start. If you have your highest priority workflow set to one, and all others set to 500, the benefits of fairness may not be significant or even noticeable. Concurrency throttling can also affect the order of execution for your workflows. Concurrency throttling takes precedence over Eligible Execution Time and Priority. A concurrency throttle might only allow a certain number of high-priority workflows to execute at once, which could cause some low-priority workflows to be executed instead (An example of this is demonstrated under “Concurrency Throttle” below). For more information on concurrency and the fairness time window, refer to Runtime Configuration and Runtime Variables.
Order of Execution Examples
Eligible Execution Time and Priority
Group A contains 15 workflows that each take 1 minute to execute. All of the workflows in Group A are submitted at 8:00. The workflows in Group A have a priority of 300.
Group B contains 10 workflows that each take 2 minutes to execute. All of the workflows in Group B are submitted at 8:05 and have a priority of 100.
The workflows in both groups are ready and scheduled for immediate execution. The concurrency throttle is set to 1, meaning only 1 workflow may execute at a given time.
At 8:00, Group A would have preference and begin executing. At 8:05, 5 workflows from Group A have already been executed when Group B is submitted to the engine.
Now, both groups are scheduled for immediate execution, since the wall clock (the current time) has advanced beyond the Eligible Execution Time for both groups. Consequently, either group may execute next, but because Group B has a higher priority, Group B’s workflows begin executing, since its priority is higher than Group A’s.
Once all 10 of Group B’s workflows have run to completion, the remaining 10 workflows from Group A will execute.
Workflow Creation Timestamps
Group A and Group B both contain 10 workflows that take 1 minute to execute. The workflows in both groups have a priority of 300, and the concurrency throttle is set to 1.
Group A’s workflows are submitted to the engine at 8:00 and therefore have an Eligible Execution Time of 8:00. Group B’s workflows are submitted at 8:05 and therefore have an Eligible Execution Time of 8:05.
At 8:00, Group A would have preference and begin executing. At 8:05, Group B’s workflows are submitted to the engine.
This time, precedence goes to the workflows with earlier Eligible Execution Times. In this case, because Group A was submitted at 8:00. The remaining 5 workflows from Group A must finish executing before any workflows in Group B will run.
Group A contains two workflows, each of which takes 10 minutes to complete. These workflows have a priority of 100, with a concurrency throttle of 1 on this branch in the workflow namespace. This means that only 1 workflow from Group A may execute at a given time.
Group B contains three workflows that each take 1 minute to complete. These workflows have a priority of 300, with a concurrency throttle of 1 on this branch of the workflow namespace.
Both groups of workflows are submitted at 8:00. The concurrency throttle on the root node of this workflow namespace is set to 2, meaning that only 2 workflows can execute at the same time.
At 8:00, the first workflow from Group A and the first workflow from Group B will begin executing simultaneously. This is the effect of concurrency throttling: while 2 workflows are allowed to run simultaneously, each group is throttled so that only 1 workflow can execute from that group.
Because concurrency throttling will not allow the second workflow from Group A to execute while the first is still running, all three workflows from Group B will finish before the second workflow from Group A runs, even though Group A’s workflows would normally have preference due to their priority.
Group A and Group B both contain two workflows that execute for a minute, wait at a trigger for two minutes, and then finish executing for one more minute. The workflows in both groups have the same priority, and the concurrency throttle is set to 1.
Group A’s workflows are submitted to the engine at 8:00 and therefore have a workflow creation timestamp of 8:00. Group B’s workflows are submitted at 8:01 and therefore have a workflow creation timestamp of 8:01.
At 8:00, Group A would have preference and begin executing. At 8:01, Group B’s workflows are submitted to the engine.
This time, precedence goes to the workflows with earlier workflow creation times, and a workflow in Group A begins executing. At 8:01, the second workflow in Group A executes. At 8:02, the first workflow in Group B can execute since both workflows in Group A are waiting at their triggers. At 8:03, a workflow in Group A as well as Group B is eligible to run. However, since the Group A workflows have older workflow creation timestamps, a Group A workflow must run next.
This FIFO Scheduling scenario shows that among those workflows that have Eligible Execution Times at or beyond the wall clock time (the current time), they begin executing according to the order of their workflow creation timestamps. If the concurrency throttle is 1, they execute in a true queue-like manner. If the concurrency throttle is more than 1, they begin executing in FIFO order, but the first workflow may not finish before a successive workflow starts. Finally, a workflow that begins executing and reaches a trigger must wait until the workflow’s eligible execution time once again reaches the wall clock time, at which time that workflow is subject to FIFO Scheduling once again.