TASIOMIND.DEV — OPERATIONAL▸▸▸FULL STACK DEVELOPER @ GWQ SERVICEPLUS AG▸▸▸FOUNDER — K8SGPT.AI▸▸▸OPEN SOURCE: ACTIVE▸▸▸DISTRIBUTED SYSTEMS / KUBERNETES / AI▸▸▸RUST + GO + PYTHON▸▸▸FIELD TESTED / STATUS — NOMINAL▸▸▸LOCATION: EUROPE/BERLIN▸▸▸TASIOMIND.DEV — OPERATIONAL▸▸▸FULL STACK DEVELOPER @ GWQ SERVICEPLUS AG▸▸▸FOUNDER — K8SGPT.AI▸▸▸OPEN SOURCE: ACTIVE▸▸▸DISTRIBUTED SYSTEMS / KUBERNETES / AI▸▸▸RUST + GO + PYTHON▸▸▸FIELD TESTED / STATUS — NOMINAL▸▸▸LOCATION: EUROPE/BERLIN▸▸▸
← sysadmin

Use rsync to mirror websites

September 6, 2017

tl;dr

code
apt update && apt install rsync

#Access via remote shell:
#  Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
#  Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

# Pull: sync remote with local
rsync -vhaze ssh user@server.example.com:/var/www/ /var/www
rsync -vPhaze "ssh -p 1234 -i /root/.ssh/id_rsa" root@server.example.com:/var/www/cakebox.me/public_html/ /var/www/cakebox.me/public_html
rsync -qaze "ssh -p 1234 -i /root/.ssh/id_rsa" root@server.example.com:/etc/letsencrypt/archive :/etc/letsencrypt/live :/etc/letsencrypt/renewal /etc/letsencrypt/
code
-v, --verbose               increase verbosity
-h, --human-readable        output numbers in a human-readable format
-q, --quiet                 suppress non-error messages
-a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
-z, --compress              compress file data during the transfer
-e, --rsh=COMMAND           specify the remote shell to use
    --progress              show progress during transfer
-P                          same as --partial --progress
  • explainshell for the rsync command.
  • -e is basically the shell to use, e.g. ssh -p 2234 -i /user/.ssh/id_rsa, followed by the command
  • -a is used for preservation (ownership, permissions, soft links etc.)
  • You can specify custom SSH port and SSH key to use for the connection. The path for the key file needs to be absolute path, ~ will not expand, so you have to use /userhome
  • Make sure which way you're syncing. If note sure, you could end up overwriting imprtant files. For example config files with database connection details for example, this has happened.
  • An easy way to determine which way you are syncing is the -e flag. -e determines the remote shell, so if you're using the option, you're pulling, i.e. syncing remote changes to local.
  • -e can not be used when you're syncing local changes to remote

Create an unprivileged user for the sake of transferring files

code
useradd -d /home/rsyncuser -m -s /bin/bash rsyncuser
passwd rsyncuser

Create and copy an SSH key for password-less access

code
# Generate, copy and connect with an SSH key
ssh-keygen -t rsa
ssh-copy-id -i /home/
rsyncuser/.ssh/id_rsa.pub rsyncuser@webserver.example.com
ssh rsyncuser@webserver.example.com

Create a cronjob to automate the whole thing

code
crontab –e
*/5 *  *  * * rsync -vhaze "ssh -p 1234 -i /root/.ssh/id_rsa" root@server.example.com:/var/www/cakebox.me/public_html/ /var/www/cakebox.me/public_html

Links