Resume scp/rsync file transfer

This is very basic Linux knowledge, but the question has arisen too often and I decided to document it.

Problem:

Transfer (or upload/download) a large tree of files and directories from one machine to another using a ssh connection, and being able to resume/continue if the operation is interrupted.

Solution:

Until now, I believe that the best solution is using rsync over ssh, since rsync has a feature to resume interrupted file transfer, even when an entire tree of directories is involved.

Command line:

rsync -vrlPtz -e ssh host:/remote_path/* /local_path/

Explained:

-e ssh rsync will use ssh client instead of rsh
-z compress file transfer
-t preserve time (other attributes as owner or permissions are also possible)
-l copy symlinks as symlinks
-P resume incomplete file transfer
-r recursive into subdirectories
-v verbose
Optionally, you may also add following options, that make sense only if both machines have the save user name space:
-p preserve permissions
-o preserve owner
-g preserve group

10 Responses to Resume scp/rsync file transfer

  1. Lucas says:

    Good stuff Daniel! Even though it is basic knowledge, keeping up with several man pages fighting for space in your head is hard, so some tips once in a while are very helpful!

    See ya buddy,

  2. Phil LaPier says:

    Thanks Daniel.
    You saved me so much hassle with this helpful tip. Apparently filezilla has a bug in it so that when you are resumeing a file larger than 2GB or so, it messes up the filesize and checksum – leaving the file corrupt. Again, thank you.

  3. Pingback: ssh, you touch me in all the right ways. | Phil LaPier

  4. Grant says:

    Good info. Thanks!

  5. Dan says:

    Hi Daniel.
    I get the following answer when executing the command.

    sent 16 bytes received 62 bytes 7.43 bytes/sec
    total size is 1350389760 speedup is 17312689.23

    and it stops there. Any idea?

    Thank you

    Dan

  6. Pingback: nottingham web hosting

  7. Pingback: state flags

  8. Pingback: Cheap Web Hosting and Domain Names

  9. md says:

    Completely wrong, and led me into trouble :(
    rsync 3.0.6
    -P is –progress –partial, meaning “keep partial files after transfer”. However, it does *not* resume transfer – every new transfer deletes the partial file and starts from scratch.

    You need to use –append to resume interrupted transfer.

  10. phd says:

    lol md is right
    Daniel Ferbers you should edit youe post!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: