Speed up network transfers with ssh and tar

When you use sftp/scp to transfer directories, you notice it ends up taking a long time. It works on files one at a time and they can often take a small amount of time to start and finish. A solution is to use tar to stream the file over ssh (so you have one continuious stream, without starting/stopping a lot).

ssh [email protected] “tar cf – big_dir/” | tar xf –

I timed transferring a 330MB directory with more than 3600 files in it.

Transferring over 10/100 Ethernet over SMBFS with ‘cp -r’

real    2m45.495s
user    0m0.100s
sys     0m4.424s

Transferring over 54mb WiFi (at 70% signal strength) using ssh/tar

real    2m8.800s
user    0m12.377s
sys     0m10.501s

Transfering over 10/100 Ethernet using ssh/tar

real    0m37.929s
user    0m5.628s
sys     0m4.068s

It took 37 less seconds over wireless (pretty impressive, especially when dealing with only 330MB). It was a full 2 MINUTES (and 8 seconds) faster over the same kind of connection.

This is useful both in a LAN enviornment and over the internet.

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.