Resume scp/rsync file transfer

November 19, 2008

This is very basic Linux knowledge, but the question has arised too often and I decided to document it.

Problem:

Transfer (or upload/download) a large tree of files and directories from one machine to another using a ssh connection, and being able to resume/continue if the operation is interrupted.

Solution:

Until now, I believe that the best solution is using rsync over ssh, since rsync has a feature to resume interrupted file transfer, even when an entire tree of directories is involved.

Command line:

rsync -vrPtz -e ssh host:/remote_path/* /local_path/

Explained:

-e ssh rsync will use ssh client instead of rsh
-z compress file transfer
-t preserve time (other attributes as owner or permissions are also possible)
-P resume incomplete file transfer
-r recursive into subdirectories
-v verbose

Entry Filed under: Linux, planetLTC. .

5 Comments Add your own

  • 1. Lucas  |  November 21, 2008 at 5:33 pm

    Good stuff Daniel! Even though it is basic knowledge, keeping up with several man pages fighting for space in your head is hard, so some tips once in a while are very helpful!

    See ya buddy,

    Reply
  • 2. Phil LaPier  |  December 3, 2008 at 7:13 am

    Thanks Daniel.
    You saved me so much hassle with this helpful tip. Apparently filezilla has a bug in it so that when you are resumeing a file larger than 2GB or so, it messes up the filesize and checksum – leaving the file corrupt. Again, thank you.

    Reply
  • 3. ssh, you touch me in all the right ways. | Phil LaPier  |  December 3, 2008 at 11:06 pm

    [...] thanks to Daniel Ferbers, I managed to transfer the file with success and without reading through lots of man [...]

    Reply
  • 4. Grant  |  August 20, 2009 at 5:26 pm

    Good info. Thanks!

    Reply
  • 5. Dan  |  August 27, 2009 at 12:51 pm

    Hi Daniel.
    I get the following answer when executing the command.

    sent 16 bytes received 62 bytes 7.43 bytes/sec
    total size is 1350389760 speedup is 17312689.23

    and it stops there. Any idea?

    Thank you

    Dan

    Reply

Leave a Comment

Required

Required, hidden

Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <pre> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Trackback this post  |  Subscribe to the comments via RSS Feed


Disclaimer

This is the technical weblog of Daniel Felix Ferber. The postings on this site are his own and don’t necessarily represent neither IBM’s, Stefanini IT Solutions nor Petrobras positions, strategies or opinions.

My Personal Weblog

This weblog is dedicated for my technical articles written in English. If you are interested in my personal thoughs, or my articles in Portuguese, please visit Daniel Ferbers Weblog.

Blogroll

Feeds

Pages

 

November 2008
M T W T F S S
« Oct   Mar »
 12
3456789
10111213141516
17181920212223
24252627282930

Archives

Meta