The purpose of this document is to provide information on software that can increase the speed of transferring large files between trading parnets.

This document covers the use of sofware that is designed to provide functions for transferting files faster than the classic file transfer protocols such as ftp. This document provides useful resources to help with the implementation of accelerated file transfers protocols.

Accelerated File Transfer Protocol is a software technology that improves the time it takes to transfer data between end-points.  It is faster than traditional file transfer protocols such as FTP.

Instead of treating large files like just one item, the AFT deals with each individual byte by extracting and transmitting only the difference between new and old versions instead of transmitting the whole data. This way, if there is an interruption in the transfer or some of the data is corrupted, the AFT can pick up where it left off without affecting the already downloaded content and can reach out and grab only the data gaps that are missing or being replaced.

A possible benefit could be the transmission and updates of large files in the Automotive Industry, such as Parts Master, and Labor Operations data.

We have evaluated two software products in this document:  

JScape AFTP is a commercial product and Rsync is an open source product.

In the next section we will look at the capabilities of each product.  

Table 1. Product Features

Rsync

Rsync is a software application and network protocol for Unix-like systems with ports to Windows that synchronizes files and directories from one location to another while minimizing data transfer by using delta encoding when appropriate.

Rsync uses the 'rsync algorithm’, which provides a very fast method for bringing remote files into sync. An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction. Rsync can copy or display directory contents and copy files, optionally using compression and recursion.

In daemon mode, rsync listens on the default TCP port of 873, serving files in the native rsync protocol or via a remote shell such as RSH or SSH. In the latter case, the rsync client executable must be installed on the remote machine as well as on the local machine.

JScape AFT

JSCAPE offers accelerated file transfer capabilities in its managed file transfer (MFT) software utilizing AFTP.  AFTP (Accelerated File Transfer Protocol) is a hybrid TCP/UDP protocol developed by JSCAPE that enables users to accelerate file transfers over high-speed networks that are unable to fully utilize network throughput due to high latency and/or packet loss. 

Under these conditions JSCAPE software can accelerate file transfers up to 100 times faster than FTP and other file transfer protocols

A series of file transfers were performed to compare the transfer rates between these products. The tables below summarize the findings, following by a detailed description of each file transfer step. We have written down our observations, after comparing the performance between each product.

EVALUATION ENVIRONMENT

Amazon EC2 Instance

MacBook

Internet Connectivity

Data Used for Testing

Evaluation Results

 Table 1. Uncompressed file transfer

Table 2. Compressed file transfer

Table 3. File transfer after increasing the file size by 1.1 MB

EVALUATION STEPS

Step 1 - SFTP Transfer

Uncompressed file

Direction: Push to Amazon EC2 Instance

Duration: 23 minutes and 24 seconds

File Size: 317.9 MB

Compressed file using the gzip tool

Direction: Push to Amazon EC2 Instance

Duration: 1 minute and 2 seconds

File Size: 1.4 MB

Step 2 - Rsync Transfer

Uncompressed file

Direction: Push to Amazon EC2 Instance

Direction: 23 minutes and 39 seconds

File Size: 317.9 MB

Compressed file with Rsync native compression

Direction: Pusth to Amazon EC2 Instance

Duration: 1 minute and 2 seconds

File Size: N/A when using native compression

Step 3 - JScape Transfer

Uncompressed file

Direction: Pusth to Amazon EC2 Instance

Duration: 1 minute and 11 seconds

File Size: 317.9 MB

Compressed file using the gzip tool

Direction: Push to Amazon EC2 Instance

Direction: 1 minute and 2 seconds

File Size: 1.4 MB

We increased the file size by 1119360 bytes and transferred the file again. Thre reason for doing this is maily to test the Rsync delta updates of the data.

Step 4 - SFTP Transfer

Direction: Push to Amazon EC2 Instance

Duration: 24 minutes and 6 seconds

File Size: 319 MB

Step 5 - Rsync Transfer

Direction: Pusth to Amazon EC2 Instance

Duration: 26 seconds

File Size: 319 MB

Step 6 - JScape Transfer

Direction: Push to Amazon EC2 Instance

Duration: 1 minute and 7 seconds

Fule Size 319 MB

OBSERVATIONS

Rsync – SFTP Comparison

JScape – SFTP Comparison

Rsync – JScape Comparison

CONLUSION

As we can see based on the evaluation steps above, Rsync and JScape AFT file transfers are much more efficient than conventional protocols such as FTP. If a business case requires the transfer of large data, you should consider utilizing one of these two tools. Rsync native compression, and delta updates can make the distribution of large files very fast, and efficient. JScape AFT can also accomplish the same tasks, and because of JScape's hybrid TCP/UDP protocol, file transfers are efficient during heavy traffic, and network congestion.

Rsync Website

Wikipedia - Rsync article

JScape Accelerated File Transfer