Tips & Tricks of the Trade: file sync

There is the ole trusty rsync

$ rsync -avz ~/compressed_file.tar.gz remoteuser@otherhost:/share/

$ ssh remoteuser@otherhost "tar zxf /share/compressed_file.tar.gz -C /share/uncompressed/"

Transfer and Extract in One Step

rsync -avz -e ssh ~/compressed_file.tar.gz remoteuser@otherhost:/dev/stdout | ssh remoteuser@otherhost "tar zxf - -C /share/uncompressed/"

Use mbuffer when dealing with large data transfers, especially over unstable networks, where buffering can help ensure smoother and more reliable data flow.

$ mbuffer -s 1K -m 512 -i ~/compressed_file.tar.gz | ssh remoteuser@otherhost "tar zxf - -C /share/uncompressed/"

• mbuffer -s 1K -m 512 -i "$SOURCE_FILE"
  • -s 1K: Sets the buffer size to 1 Kilobyte.
  • -m 512: Allocates 512 Megabytes of memory for buffering.
  • -i "$SOURCE_FILE": Specifies the input file to read from.
• |: Pipes the output of mbuffer to the next command.
• ssh remoteuser@otherhost "tar zxf - -C /share/uncompressed/"
  • Connects to the remote host otherhost with the user remoteuser.
  • "tar zxf - -C /share/uncompressed/": Extracts the tarball received from standard input (-) into the /share/uncompressed/ directory on the remote host.

Use pv when you need a simple way to monitor the progress of data through a pipeline with minimal overhead.

$ pv ~/compressed_file.tar.gz | ssh admin@169.254.x.x "tar zxf - -C /share/uncompressed/"

• pv "$SOURCE_FILE"
  • pv: Monitors the progress of data through the pipe.
  • "$SOURCE_FILE": Specifies the file to transfer.
• |: Pipes the output of pv to the next command.
• ssh admin@169.254.x.x "tar zxf - -C /share/uncompressed/"
  • Connects to the remote host with IP 169.254.x.x using the user admin.
  • "tar zxf - -C /share/uncompressed/": Extracts the tarball received from standard input (-) into the /share/uncompressed/ directory on the remote host.