Resume scp/rsync file transfer

This is very basic Linux knowledge, but the question has arisen too often and I decided to document it.

Problem:

Transfer (or upload/download) a large tree of files and directories from one machine to another using a ssh connection, and being able to resume/continue if the operation is interrupted.

Solution:

Until now, I believe that the best solution is using rsync over ssh, since rsync has a feature to resume interrupted file transfer, even when an entire tree of directories is involved.

Command line:

rsync -vrlPtz -e ssh host:/remote_path/* /local_path/

Explained:

-e ssh rsync will use ssh client instead of rsh
-z compress file transfer
-t preserve time (other attributes as owner or permissions are also possible)
-l copy symlinks as symlinks
-P resume incomplete file transfer
-r recursive into subdirectories
-v verbose
Optionally, you may also add following options, that make sense only if both machines have the save user name space:
-p preserve permissions
-o preserve owner
-g preserve group

About Trilead SSH open source project

Some time ago I have written an article about theĀ JSch open source project. However, I soon lost all my motivation when I faced, again, the code complexity required for JSch just to start a execution session, a file copy or a port forwarding with JSch. Nevertheless the completely misunderstood authentication API for JSch let me to a mood even worse. I started looking for alternatives and fortunately I found Trilead SSH, a pure Java implementation of SSH, available with the BSD license.

I have not yet tested Trilead SSH extensively to state that it provides a stable and complete SSH implementation. But results were very promising. Unfortunately, I have not much information about performance of data transfer with Trilead SSH.

I think that the major advantage of Trilead SSH is its very simple and intuitive API, directly targeted to tasks that shall be solved for someone that is using SSH. Everything is made very easy: execute a command, do port forwarding, transfer files. All complexity is done behind the scenes and a single method call uses to be enough. There is no need to care about SSH details as it was required for JSch.

In special, I liked the Trilead SSH API resposible for authentication that is very straightforward and matches very well the steps to log into the remote host. Even better, I was able to implement my own logic to respond to authentication failure exactly to the application needs. In this situation Trilead SSH is simplier and more flexible that JSch.

As a drawback, it is not possible to have low level access to the SSH protocol. However, this seems to be a very improbable requirement.

Depite of the simple API, Trilead SSH claims to implement all major ssh features: sessions (remote command execution and shell access), X11 forwarding, local and remote port forwarding, local stream forwarding, SCP and SFTP. Many options are available for authentication methods and cipher types.

The API is very well documented and includes examples and advices, what is very rare on a typical javadoc.

Conclusion

Replacing JSch by Trilead SSH has been a good choice, although I have not yet evidences for better performance nor stability. But the code become considerable simpler and shorted. And development time was reduces many times.