When you use sftp/scp to transfer directories, you notice it ends up taking a long time. It works on files one at a time and they can often take a small amount of time to start and finish. A solution is to use tar to stream the file over ssh (so you have one continuious stream, without starting/stopping a lot).
ssh [email protected] “tar cf – big_dir/” | tar xf –
I timed transferring a 330MB directory with more than 3600 files in it.
Transferring over 10/100 Ethernet over SMBFS with ‘cp -r’
real 2m45.495s
user 0m0.100s
sys 0m4.424s
Transferring over 54mb WiFi (at 70% signal strength) using ssh/tar
real 2m8.800s
user 0m12.377s
sys 0m10.501s
Transfering over 10/100 Ethernet using ssh/tar
real 0m37.929s
user 0m5.628s
sys 0m4.068s
It took 37 less seconds over wireless (pretty impressive, especially when dealing with only 330MB). It was a full 2 MINUTES (and 8 seconds) faster over the same kind of connection.
This is useful both in a LAN enviornment and over the internet.