Must-Haves for Trouble-Free File Transfers
When you embark on a new managed file transfer project or you just want to review your existing file transfer processes, here are 5 things for you to review in order to make your system more trustworthy to everyone involved and to create smoothly running processes that require minimal human intervention.
1. Confirm Your File Transmissions
Everyone wants to know that their files arrived successfully. Your company, your vendors, your partners — everybody. An explicit confirmation is great. A simple way to do that is to have the receiver generate a feedback file to confirm that everything was received correctly. This feedback file can be sent back to the sender by any way that makes sense:
- Placed on the sender’s FTP server
- Sent by email
- Delivered via a Web Service invocation
- Placed onto a message queue
2. Transmission Failures and Notices
What if a file transfer fails? If a file literally cannot be sent from one FTP server to another, the failure could be for common reasons such as:
- Network problems
- Server down
- Unable to login
- Path to file doesn’t exist
- Insufficient permissions to access file A common best practice is to soft fail for a problem above, wait one hour, and then retry automatically, without any human intervention. After five soft failures, a hard failure causes a failure notification to be sent to both the sending FTP server’s staff as well as the receiving FTP server’s staff.
3. Files Fail Verification
Files can transfer successfully, but they don’t arrive in the expected format. For example, a zip file might be corrupted or there might be some missing files from an uncorrupted zip file.
In this failure scenario, soft and hard failures don’t apply.
Common best practices include:
- On the receiving side, verify files after reception.
- On the sending side, verify large files before transmission.
- Immediately notify both sending and receiving staff in case of verification failure.
- Sometimes, when verification fails from the sending side, notification to the receiving side can be delayed if it’s possible that the problem can be fixed before the transmission deadline.
4. Best Protocol
Oftentimes you are constrained as to what file transfer protocol to use. One side or the other may only support one protocol and be unable or unwilling to change.
However, if you do have a choice, using SFTP (SSH File Transfer Protocol) is a best practice. It’s widely used, well supported and understood, and very secure.
5. Using Dates and Timestamps in File Naming Conventions
New data files are usually generated daily, even hourly. To avoid confusion and to prevent older files from being overwritten with current data, you need to keep these data sets cleanly separated from each other. Use dates and timestamps embedded in your file names to achieve this goal.
- 2011-10-18-sales.zip (sales data for 18 Oct 2011)
- 18-Oct-2011-sales.zip (a different date format)
- 18-Oct-2011-09-00.zip (include hourly sales data for 09:00)
- monday-morning-sales.zip (Monday morning’s sales data)
- 2011-10-18-09-45.zip (sales data for 09:45 on 18 Oct 2011)
Use these 5 best practices to create a robust managed file transfer environment for your organization and your file trading partners. These 5 things offer a simpler environment, allowing your managed file transfer operations to be efficiently integrated with other systems, applications, and databases. These 5 best practices have proven successful in saving valuable engineering bandwidth and time for your operations staff.