Call Sales +1 (720) 398-5435 | Email Sales sales@flux.ly — Call Support +1 (702) 879-4130 | Email Support support@flux.ly

Job Scheduling and File Transfer

Making Reusable File Transfer Workflows

reusable file transfer workflow flux

How does Flux make file transfer workflows easier?

A Frequent Question

An example of a frequent query (edited here for clarity) received by our technical support staff:

“… we have a lot of file transfers that all implement the same logic. So we’d like to design this logic once and then just feed the transfer details as parameters for runtime instances of the same design. I get that Runtime Configurations would be the way to go about implementing this. Then adding a new transfer of the same type would simply be a task of defining a Runtime Configuration document and somehow triggering a workflow using this document, right? Basically we have three types of source/targets: SFTP, UNC or SCP (copy to Linux mount). A transfer can be between any two of these. In the example below we have SFTP to UNC transfer.

The goal is to implement the following 2 requirements:

  1. Implement the aforementioned above.
  2. A bit more complex workflow where we would first transfer the file and if successful then trigger some PowerShell task for example.”

Flux’s Answer – Namespaces and Runtime Variable Substitution

Flux provides two key capabilities to support these common use cases. These are:

  • Workflow namespaces to organize Flux repository workflow templates and running workflows on the Flux engine into categories of shared functionality or common runtime parameters
  • Runtime variables and runtime variable substitution that configure workflows submitted to specific namespaces on the Flux engine for a specific use

Workflow namespaces are virtual folders in which to group similar kinds of workflows or workflows that need a common set of configuration properties. Namespaces can be leveraged differently on the Flux repository and the Flux engine. Workflows can be stored in the Flux repository under one set of namespaces (e.g., by functionality such as capital budgets or ACH payments) and submitted to the Flux engine under a different namespace (e.g. daily runs or a specific customer run).

Runtime variables separate the specific values a workflow requires from the workflow itself. For example, you can separate variables such as server names or file locations from the workflow itself. Runtime variables are saved in the Flux engine’s runtime configuration properties file. It is common for Flux users to have different runtime configuration files for different environments, and place these under version control. For instance, production, QA, and developer instances of Flux may all utilize the same workflows, but with different runtime configuration files so that each workflow runs with different values depending upon the environment (i.e., namespace) the workflow is run within.

Working the Example

For a Flux workflow to accomplish requirement (1) above involves defining a Flux file copy action that is runtime configurable. Flux file copy actions contain the source and target definitions for generic file copies between desired sources and destination. Shown below is a file copy action that copies from a UNC Server to an SFTP server. Note that the Host Name has a value ${runtime UNCServer}. The ${runtime VARIABLE_NAME} notation refers to a variable that will be resolved at the time the workflow executes. When the workflow is submitted and this file copy action is initiated, the Flux engine will lookup every instance of ${runtime VARIABLE_NAME} and substitute it with a value from the runtime configuration properties files maintained in the Flux server (and editable from the Flux user interface). Values are substituted by reading the runtime configuration hierarchically.

file copy action within flux workflows

The above workflow depiction (just a single Flux file copy action) shows all that is required to move a file from one location to another. The definitions for the file copy action properties are shown below.

flux file sources

flux transfer file targets

Other file copy action workflows may be configured as follows:

  • SFTP to UNC with workflow saved as (/Template/SFTP to UNC)
  • SFTP to Localhost (Local file system) with workflow saved as (/Template/SFTP to Localhost)
  • LocalHost to SFTP with workflow saved as (/Template/Localhost to SFTP)
  • Localhost to UNC with workflow saved as (/Template/Localhost to UNC
  • UNC to Localhost with workflow saved as (/Template/UNC to Localhost)
  • UNC to SCP with workflow saved as (/Template/UNC to SCP) (Using Flux’s SSH Download action for SCP)
  • SCP to UNC with workflow saved as (/Template/SCP to UNC) (Using Flux’s SSH Upload Action for SCP)

Each one of these file copy action workflows would be saved as separate repository workflows into the /Template/ Workspace for later reuse.

To then use these workflows with parameters assigned at runtime, the following properties would be defined into Flux’s runtime configuration properties. Sample values are shown (after the = sign) for illustration purposes.

Note: As a safety precaution and general practice, the root namespace (those properties with only a single slash) has no values after the ‘=’ sign. This is to ensure that the template workflows themselves will not execute successfully if submitted into a namespace that has no runtime configuration variables specified for it. Since there are no values specified for the /Template namespace, workflows submitted would inherit from the root namespace, and since no values are specified in the root namespace, template workflows would fail.

# SFTP Server and Password and user names for the root namespace. Note that after the = nothing is defined
/SFTPServer=
/SFTPServerUserName
=
/SFTPServerPassword
=
/SFTPServerPort
=22

/SFTPServer=
/SFTPServerUserName
=
/SFTPServerPassword
=
/SFTPServerPort
=22

# SFTP Server and Password and user names for specific customers
/Customer1/SFTPServer=doc.flux.ly
/Customer1/SFTPServerUserName
=sfluxly
/Customer1/SFTPServerPassword
=*PASSWORD*
/Customer1/SFTPServerPort
=22

/Customer2/SFTPServer
=s.flux.ly
/Customer2/SFTPServerUserName
=flux
/Customer2/SFTPServerPassword
=*PASSWORD*
/Customer2/SFTPServerPort
=22

# UNC Server and Password and user names for root namespace
/UNCServer=
/UNCServerUserName
=
/UNCServerPassword
=
/UNCServerDomain
=
/UNCServerPort
=
/UNCServerShare
=

/Customer1/UNCServer=flux-support
/Customer1/UNCServerUserName
=support
/Customer1/UNCServerPassword
=*PASSWORD*
/Customer1/UNCServerDomain
=flux-support
/Customer1/UNCServerPort
=133
/Customer1/UNCServerShare
=/uncshare/

/Customer2/UNCServer=flux-rcs-2
/Customer2/UNCServerUserName
=support
/Customer2/UNCServerPassword
=*PASSWORD*
/Customer2/UNCServerDomain
=flux-support
/Customer2/UNCServerPort
=133
/Customer2/UNCServerShare
=/uncshare/

When needed, any of the template workflows can be selected and submitted to the Flux engine in the desired namespace to perform a file action based on that namespace’s runtime variables. So the workflow /Template/SFTP to UNC could be selected from the repository and submitted to the engine as /Customer1/SFTP to UNC. The file copy would then use the runtime properties defined for Customer1 to perform the file copy.

To support requirement (2) above (i.e., perform a file copy and if successful run a Powershell script), a workflow is defined that contains a Flux FlowChart Action flowing into a Flux Process Action. This workflow runs the template file copy workflow synchronously (i.e., until it completes) before letting the Powershell command run. This workflow is also runtime configured to make it reusable. Its design would look like this:

file copy then run powershell command workflow
file copy action configuration flux

The above workflow depiction illustrates running a Powershell script after completing a file copy. The Run the Powershell Command action can also be configured with a runtime variable to select which Powershell Command to execute based on the namespace the workflow is running within.

Configurable Error Handling

In addition to having the workflows configured at runtime, errors encountered can also be dealt with via runtime properties. A common solution is a default error handler defined to mail exceptional conditions and error messages to support staff in the event a workflow encounters issue.

error handling flux workflow

The above workflow depiction shows /Demo/ErrorHandler with Mail Action and Configurable Restart, which is used to notify and restart failed workflows in the /Demo/ namespace. The required runtime variables are defined below. Note that in this case, the mail actions does use values from the “/” root namespace (against the best practice mentioned above), since generally only one mail server is available.

# ————————————
# Mail Server setups for IMAP and SMTP
# ————————————
/MailActionSMTPServer1Username=supportAccount@flux.ly
/MailActionSMTPServer1Password=***PASSWORD***
/MailActionSMTPServer1Host=smtp.gmail.com
/MailActionSMTPServer1Port=587
/MailActionSMTPServer1SSL=true
/MailActionSMTPServer1Protocol=SMTP
/MailActionSMTPServer1FromAddress=support@flux.ly
/MailActionSMTPServer1ToAddress=support@flux.ly

/RESTART=false

Summary

Reusing workflows make for easier maintenance and quicker delivery of file transfer automation. Flux’s namespace support in conjunction with its substitution of variables at runtime simplifies the delivery of such reusable flows while still providing specific and unique parameters for different processing needs.

Top