Various methods to improve rsync speeds
2 min read
rsync is a command line tool to transfer data between a source and a destination. A destination or source can be remote.
Avoid the following:
- -z - try to avoid using compressing network traffic.
- --append-verify - resume an interrupted transfer. This sounds like a good idea, but it has the dangerous failure case: any destination file the same size (or greater) than the source will be IGNORED. Also, it checksums the whole file at the end, meaning no significant speed up over --no-whole-file while adding a dangerous failure case.
Use the following in the command:
- -a - Archive mode - recursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.
- -v - Increase verbosity
- -A - Preserves ACLs
- -H - Preserves hard-links
- -X - Preserves extended attributes
- -W - Copy files whole - always use this if you don't want it to compare differences;
rsyncis designed to compare differences and only update the changes. Remove this if you require the compare. - -S - Handles sparse files (files that are for instance holding a file system and that just take the space needed by their actual size)
- --no-compress - This may have a big impact on speed gain.
- --exclude-from - Exclude files you might not need.
$ rsync -avAXEWSlHh /source /destination --no-compress --info=progress2 --dry-run
Test using --dry-run, then remove this to execute the sync.
