Backup

Table of Contents

1 Reformat HDD

sudo fdisk /dev/sda

Format the drive in a way that you like.

To create an Ext4 file system on one of the partitions:

sudo mkfs.ext4 /dev/sda1

Set label on partition.

sudo e2label /dev/sda1 "2TBext4"

Create proper permissions on the partition.

chown -r <USER>:<GROUP> /path/to/disk

2 Mouting a external USB disk

Find your disk id:

sudo blkid

Add to /etc/fstab

# Intenso 2TB
UUID=a3c930e1-c549-4755-9afd-609396225988       /media/backup   ext4            defaults,nofail,x-systemd.device-timeout=1      0       2

3 Rsync

rsync -ave ssh local/
rsync -avze ssh --progress local/ arch:/ --exclude=/Library

--archive
-v
-z

3.0.1 Arch full system

https://wiki.archlinux.org/index.php/full_system_backup_with_rsync

This is used to the back up Arch to OpenWRT.

sudo rsync -av -e ssh --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / router:/mnt/2TBext4/Backups/Arch
sudo rsync -av --progress --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/boot/*","/home/nils/.cache/*"} / router:/mnt/2TBext4/Backups/Arch
sudo rsync -av --progress --numeric-ids -e "ssh -T -x" --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/boot/*","/home/nils/.cache/*"} / router:/mnt/2TBext4/Backups/Arch

To run a script that will run to completion.

sudo rsync -av --progress --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/boot/*","/home/nils/.cache/*"} / router:/mnt/2TBext4/Backups/Arch
while [ !$ -ne 0 ];
do
    sudo rsync -av --progress --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/boot/*","/home/nils/.cache/*"} / router:/mnt/2TBext4/Backups/Arch
done

4 Arch backup

Install dependencies

sudo pacman -S nfs-utils
sudo systemctl start nfs-client.target

Try to mount it manually

sudo mount --verbose nas:/backup-dell-arch -overs=3 /media/backup/

Edit /etc/fstab

nas:/backup-dell-arch   /media/backup  nfs          noauto,x-systemd.automount,x-systemd.device-timeout=10,timeo=14,x-systemd.idle-timeout=1min 0 0

Resources:

5 QNAP TS-231

5.1 NFS Shares

Share the mount and be sure to fix the NFS share. Set IP addresses!

[~] # cat /etc/exports
"/share/CACHEDEV1_DATA/backup" 192.168.1.100(rw,async,no_subtree_check,insecure,no_root_squash)
"/share/CACHEDEV1_DATA/backup-other" *(rw,async,no_subtree_check,insecure,no_root_squash)

5.2 Diagnostics

Get device info(source):

https://forum.qnap.com/viewtopic.php?f=25&t=102330&p=600056#p600056 https://forum.qnap.com/viewtopic.php?f=50&t=120037&p=596307&hilit=ts+231#p596307

https://helpdesk.qnap.com/index.php?/Tickets/Submit/RenderForm

#!/bin/sh
rm -f /tmp/nasreport
touch /tmp/nasreport
chmod +x /tmp/nasreport
cat << EOF >> /tmp/nasreport
#!/bin/sh
#
# NAS Report by Patrick Wilson
# see: http://forum.qnap.com/viewtopic.php?f=185&t=82260#p366188
#
#
echo "[code]"
echo "*********************"
echo "** QNAP NAS Report **"
echo "*********************"
echo " "
echo "NAS Model:      \$(getsysinfo model)"
echo "Firmware:       \$(getcfg system version) Build \$(getcfg system 'Build Number')"
echo "System Name:    \$(/bin/hostname)"
echo "Workgroup:      \$(getcfg system workgroup)"
echo "Base Directory: \$(dirname \$(getcfg -f /etc/config/smb.conf Public path))"
echo "NAS IP address: \$(ifconfig \$(getcfg network 'Default GW Device') | grep addr: | awk '{ print \$2 }' | cut -d: -f2)"
echo " "
echo "Default Gateway Device: \$(getcfg network 'Default GW Device')"
echo " "
ifconfig \$(getcfg network 'Default GW Device') | grep -v HWaddr
echo " "
echo -n "DNS Nameserver(s):"
cat /etc/resolv.conf | grep nameserver | cut -d' ' -f2
echo " "
echo " "
echo "HDD Information:"
echo " "
alpha='abcdefghijklmnopqrstuvwxyz'
drives=\$(getcfg Storage 'Disk Drive Number')
for ((i=1;i<=drives;++i)) ; do
   echo -n "HDD\$i -"
   if [ ! -b /dev/sd\${alpha:\$i-1:1} ] ; then
      echo "Drive absent"
   else
      hdparm -i /dev/sd\${alpha:\$i-1:1} | grep "Model"
      echo " "
      parted /dev/sd\${alpha:\$i-1:1} print
      echo " "
      /sbin/get_hd_smartinfo -d \$i
      echo " "
   fi
done
echo "Volume Status"
echo " "
mdadm -D /dev/md0 /dev/md1 2>/dev/null
echo " "
cat /proc/mdstat
echo " "
echo "Disk Space:"
echo " "
df -h | grep -v qpkg
echo " "
echo "Mount Status:"
echo " "
mount | grep -v qpkg
echo " "
#echo "Contents of \$(dirname \$(getcfg -f /etc/config/smb.conf Public path)):"
#echo " "
#ls -lF \$(dirname \$(getcfg -f /etc/config/smb.conf Public path))/
#echo " "
#echo "Windows Shares:"
#echo " "
#for i in \$(grep \] /etc/config/smb.conf | sed 's/\[//g' | sed 's/\]//g' | grep -v global) ;do
#   echo -n "\$i:"
#   testparm -s -l --section-name=\$i --parameter-name=path 2>/dev/null
#done
#echo " "
#echo "QNAP Media Scanner / Transcoder processes running: "
#echo " "
#/bin/ps | grep medialibrary | grep -v grep
#echo " "
#echo -n "MediaLibrary Configuration file: "
#ls -alF /etc/config/medialibrary.conf
#echo " "
#echo "/etc/config/medialibrary.conf:"
#cat /etc/config/medialibrary.conf
echo " "
echo "Memory Information:"
echo " "
free | grep -v cache:
echo " "
echo "NASReport completed on \$(date +'%Y-%m-%d %T') (\$0) [/code]"
echo " "
EOF
sleep 2
clear
/tmp/nasreport
#done

6 Autofs

In my case I am using Autofs to mount a NFS share.

6.2 Installation

Add the following to: /etc/hosts. Change the IP and nas name to reflect your setup.

192.168.1.200     nas.localdomain         nas

Install Autofs

pacman -S autofs

Start and enable rpcbind

systemctl start rpcbind
systemctl enable rpcbind

6.3 Configuration

Uncomment in /etc/autofs/autofs.conf.

mount_nfs_default_protocol = 3

Add the following line to /etc/autofs/auto.master

/media   /etc/autofs/auto.nas --timeout 60

Add the following to /etc/autofs/auto.nas.

backup -rw,soft,intr,rsize=8192,wsize=8192 nas:/backup-dell-arch

Start and enable autofs.

sudo systemctl enable autofs.service
sudo systemctl start autofs.service

6.4 Useful

Show mount on nas.

showmount nas -e

Run autofs in a debug mode.

sudo systemctl stop autofs.service
sudo automount -f -v

7 Rsnapshot

“rsnapshot is a filesystem snapshot utility based on rsync. rsnapshot makes it easy to make periodic snapshots of local machines, and remote machines over ssh. The code makes extensive use of hard links whenever possible, to greatly reduce the disk space required.” / rsnapshot.org

Resources

7.1 Rsnapshot config

http://serverfault.com/questions/615783/poor-performance-executing-cp-al-over-a-nfs-export

I have made some tweaks to the rsnapshot workflow. Instead of running rm -rf and cp -al over the NFS share those commands are parsed and sent to the NAS over SSH, see custom optimization.

Also notice that exclude_file are set to an seperate file, see rsync exclude.

/etc/rsnapshot.conf

#################################################
# rsnapshot.conf - rsnapshot configuration file #
#################################################
#                                               #
# PLEASE BE AWARE OF THE FOLLOWING RULE:        #
#                                               #
# This file requires tabs between elements      #
#                                               #
#################################################

#######################
# CONFIG FILE VERSION #
#######################

config_version  1.2

###########################
# SNAPSHOT ROOT DIRECTORY #
###########################

# All snapshots will be stored under this root directory.
#
snapshot_root   /media/backup/arch

# If no_create_root is enabled, rsnapshot will not automatically create the
# snapshot_root directory. This is particularly useful if you are backing
# up to removable media, such as a FireWire or USB drive.
#
no_create_root  1

#################################
# EXTERNAL PROGRAM DEPENDENCIES #
#################################

# LINUX USERS:   Be sure to uncomment "cmd_cp". This gives you extra features.
# EVERYONE ELSE: Leave "cmd_cp" commented out for compatibility.
#
# See the README file or the man page for more details.
#
cmd_cp          /usr/local/bin/rsnapshot_custom_cmds/cp

# uncomment this to use the rm program instead of the built-in perl routine.
#
cmd_rm          /usr/local/bin/rsnapshot_custom_cmds/rm

# rsync must be enabled for anything to work. This is the only command that
# must be enabled.
#
cmd_rsync       /usr/bin/rsync

# Uncomment this to enable remote ssh backups over rsync.
#
#cmd_ssh        /usr/bin/ssh

# Comment this out to disable syslog support.
#
cmd_logger      /usr/bin/logger

# Uncomment this to specify the path to "du" for disk usage checks.
# If you have an older version of "du", you may also want to check the
# "du_args" parameter below.
#
#cmd_du         /usr/bin/du

# Uncomment this to specify the path to rsnapshot-diff.
#
cmd_rsnapshot_diff      /usr/bin/rsnapshot-diff

# Specify the path to a script (and any optional arguments) to run right
# before rsnapshot syncs files
#
#cmd_preexec    /path/to/preexec/script

# Specify the path to a script (and any optional arguments) to run right
# after rsnapshot syncs files
#
#cmd_postexec   /path/to/postexec/script


#########################################
#     BACKUP LEVELS / INTERVALS         #
# Must be unique and in ascending order #
# e.g. alpha, beta, gamma, etc.         #
#########################################

retain  hourly  24
retain  daily   7
retain  weekly  4
retain  monthly 12

############################################
#              GLOBAL OPTIONS              #
# All are optional, with sensible defaults #
############################################

# Verbose level, 1 through 5.
# 1     Quiet           Print fatal errors only
# 2     Default         Print errors and warnings only
# 3     Verbose         Show equivalent shell commands being executed
# 4     Extra Verbose   Show extra verbose information
# 5     Debug mode      Everything
#
verbose         5

# Same as "verbose" above, but controls the amount of data sent to the
# logfile, if one is being used. The default is 3.
#
loglevel        5

# If you enable this, data will be written to the file you specify. The
# amount of data written is controlled by the "loglevel" parameter.
#
logfile         /var/log/rsnapshot

# If enabled, rsnapshot will write a lockfile to prevent two instances
# from running simultaneously (and messing up the snapshot_root).
# If you enable this, make sure the lockfile directory is not world
# writable. Otherwise anyone can prevent the program from running.
#
lockfile        /var/run/rsnapshot.pid

# By default, rsnapshot check lockfile, check if PID is running
# and if not, consider lockfile as stale, then start
# Enabling this stop rsnapshot if PID in lockfile is not running
#
#stop_on_stale_lockfile         0

# Default rsync args. All rsync commands have at least these options set.
#
#rsync_short_args       -a
#rsync_long_args        --delete --numeric-ids --relative --delete-excluded

# ssh has no args passed by default, but you can specify some here.
#
#ssh_args        -o BatchMode=yes

# Default arguments for the "du" program (for disk space reporting).
# The GNU version of "du" is preferred. See the man page for more details.
# If your version of "du" doesn't support the -h flag, try -k flag instead.
#
#du_args        -csh

# If this is enabled, rsync won't span filesystem partitions within a
# backup point. This essentially passes the -x option to rsync.
# The default is 0 (off).
#
#one_fs         0

# The include and exclude parameters, if enabled, simply get passed directly
# to rsync. If you have multiple include/exclude patterns, put each one on a
# separate line. Please look up the --include and --exclude options in the
# rsync man page for more details on how to specify file name patterns.
#
#include        ???
#exclude        ???


# The include_file and exclude_file parameters, if enabled, simply get
# passed directly to rsync. Please look up the --include-from and
# --exclude-from options in the rsync man page for more details.
#
#include_file   /path/to/include/file
exclude_file    /etc/rsync_exclude.txt

# If your version of rsync supports --link-dest, consider enabling this.
# This is the best way to support special files (FIFOs, etc) cross-platform.
# The default is 0 (off).
#
link_dest       0

# When sync_first is enabled, it changes the default behaviour of rsnapshot.
# Normally, when rsnapshot is called with its lowest interval
# (i.e.: "rsnapshot alpha"), it will sync files AND rotate the lowest
# intervals. With sync_first enabled, "rsnapshot sync" handles the file sync,
# and all interval calls simply rotate files. See the man page for more
# details. The default is 0 (off).
#
#sync_first     0

# If enabled, rsnapshot will move the oldest directory for each interval
# to [interval_name].delete, then it will remove the lockfile and delete
# that directory just before it exits. The default is 0 (off).
#
#use_lazy_deletes       0

# Number of rsync re-tries. If you experience any network problems or
# network card issues that tend to cause ssh to fail with errors like
# "Corrupted MAC on input", for example, set this to a non-zero value
# to have the rsync operation re-tried.
#
#rsync_numtries         2

# LVM parameters. Used to backup with creating lvm snapshot before backup
# and removing it after. This should ensure consistency of data in some special
# cases
#
# LVM snapshot(s) size (lvcreate --size option).
#
#linux_lvm_snapshotsize 100M

# Name to be used when creating the LVM logical volume snapshot(s).
#
#linux_lvm_snapshotname rsnapshot

# Path to the LVM Volume Groups.
#
#linux_lvm_vgpath       /dev

# Mount point to use to temporarily mount the snapshot(s).
#
#linux_lvm_mountpath    /path/to/mount/lvm/snapshot/during/backup

###############################
### BACKUP POINTS / SCRIPTS ###
###############################

# LOCALHOST
backup  /home/          localhost/
backup  /etc/           localhost/
backup  /var/log/       localhost/

#backup /etc/passwd     localhost/
#backup /home/foo/My Documents/         localhost/
#backup /foo/bar/       localhost/      one_fs=1, rsync_short_args=-urltvpog
#backup_script  /usr/local/bin/backup_pgsql.sh  localhost/postgres/
# You must set linux_lvm_* parameters below before using lvm snapshots
#backup lvm://vg0/xen-home/     lvm-vg0/xen-home/

# EXAMPLE.COM
#backup_exec    /bin/date "+ backup of example.com started at %c"
#backup root@example.com:/home/ example.com/    +rsync_long_args=--bwlimit=16,exclude=core
#backup root@example.com:/etc/  example.com/    exclude=mtab,exclude=core
#backup_exec    ssh root@example.com "mysqldump -A > /var/db/dump/mysql.sql"
#backup root@example.com:/var/db/dump/  example.com/
#backup_exec    /bin/date "+ backup of example.com ended at %c"

# CVS.SOURCEFORGE.NET
#backup_script  /usr/local/bin/backup_rsnapshot_cvsroot.sh      rsnapshot.cvs.sourceforge.net/

# RSYNC.SAMBA.ORG
#backup rsync://rsync.samba.org/rsyncftp/       rsync.samba.org/rsyncftp/

Check that the config file is correct:

rsnapshot configtest

Remeber the TABs instead of spaces!

7.1.1 Rsync exclude

File with excluded directories and files.

/etc/rsyncexclude.txt

/home/nils/.IntelliJIdea*
/home/nils/.cache
/home/nils/.config/Slack
/home/nils/.config/libreoffice
/home/nils/.config/transmission
/home/nils/.dropbox
/home/nils/.eclipse
/home/nils/.ipfs
/home/nils/.java
/home/nils/.m2
/home/nils/.virtualenvs
/home/Downloads
node_modules
venv
*.pyc

7.2 Custom optimization

I have tried to run rsnapshot on a hourly basis without any custom stuff, but commands like cp -al and rm -rf take ages and a hourly backup would not be possible as those command executions takes more than a hour.

My way to solve this is to execute the commands directly on the NAS. The following 2 scripts parses the paths and change then to fit the path on the NAS. After the path parsing is done it executes it on the NAS with help of SSH.

Custom rm:

/usr/local/bin/rsnapshotcustomcmds/rm

#!/bin/bash
# This rm in only to use with rsnapshot
# The idea of this script is to minimize the time
# of the 'rm -rf' commands.
# Instead of executing it over a network share
# it will be executed on the NAS directly.

echo "Starting custom rm script..."
echo "Original path: "$*
cmd=$(sed 's@\(/media/backup/\)@/share/CACHEDEV1_DATA/backup-dell-arch/@g' <<< $*)
echo "New path: "$cmd
cmd="rm "$cmd
ssh nas $cmd
exit 0

Custom cp:

/usr/local/bin/rsnapshotcustomcmds/cp

#!/bin/bash
# This cp in only to use with rsnapshot
# The idea of this script is to minimize the time
# of the 'cp -al' commands.
# Instead of executing it over a network share
# it will be executed on the NAS directly.

echo "Starting custom cp script."
echo "Original path: "$*
cmd=$(sed 's@\(/media/backup/\)@/share/CACHEDEV1_DATA/backup-dell-arch/@g' <<< $*)
echo "New path: "$cmd
cmd="cp "$cmd
ssh nas $cmd
exit 0

7.3 Systemd service

/etc/systemd/system/rsnapshot@.service

This is the systemd service that will run rsnapshot.

[Unit]
Description=rsnapshot (%I) backup

[Service]
Type=oneshot
Nice=19
IOSchedulingClass=3
ExecStart=/usr/bin/rsnapshot %I

7.4 Systemd timers

Systemd timers are used to control scheduled snapshots.

7.4.1 Hourly timer

/etc/systemd/system/rsnapshot-hourly.timer

[Unit]
Description=rsnapshot hourly backup

[Timer]
OnCalendar=*-*-* *:45:00
Persistent=true
Unit=rsnapshot@hourly.service

[Install]
WantedBy=timers.target

Enable the timer and start it

systemctl enable rsnapshot-hourly.timer
systemctl start rsnapshot-hourly.timer
systemctl status rsnapshot-hourly.timer

7.4.2 Daily timer

/etc/systemd/system/rsnapshot-daily.timer

[Unit]
Description=rsnapshot daily backup

[Timer]
OnCalendar=*-*-* 00:30:00
Persistent=true
Unit=rsnapshot@daily.service

[Install]
WantedBy=timers.target

Enable the timer and start it

systemctl enable rsnapshot-daily.timer
systemctl start rsnapshot-daily.timer
systemctl status rsnapshot-daily.timer

7.4.3 Weekly timer

/etc/systemd/system/rsnapshot-weekly.timer

[Unit]
Description=rsnapshot weekly backup

[Timer]
OnCalendar=Mon *-*-* 00:15:00
Persistent=yes
Unit=rsnapshot@weekly.service

[Install]
WantedBy=timers.target

Enable the timer and start it

systemctl enable rsnapshot-weekly.timer
systemctl start rsnapshot-weekly.timer
systemctl status rsnapshot-weekly.timer

7.4.4 Montly timer

/etc/systemd/system/rsnapshot-monthly.timer

[Unit]
Description=rsnapshot weekly backup

[Timer]
OnCalendar=*-*-01 00:00:00
Persistent=yes
Unit=rsnapshot@monthly.service

[Install]
WantedBy=timers.target

Enable the timer and start it

systemctl enable rsnapshot-monthly.timer
systemctl start rsnapshot-monthly.timer
systemctl status rsnapshot-monthly.timer

7.5 Logs

/var/log/rsnapshot

Check hourly start and complete

egrep "/usr/bin/rsnapshot hourly: started|/usr/bin/rsnapshot hourly: completed successfully" /var/log/rsnapshot | grep -v "/usr/bin/logger" | awk '{print $1, $3, $4}'

Check daily start and complete

egrep "/usr/bin/rsnapshot daily: started|/usr/bin/rsnapshot daily: completed successfully" /var/log/rsnapshot | grep -v "/usr/bin/logger" | awk '{print $1, $3, $4}'

Check weekly start and complete

egrep "/usr/bin/rsnapshot weekly: started|/usr/bin/rsnapshot weekly: completed successfully" /var/log/rsnapshot | grep -v "/usr/bin/logger"

Check monthly start and complete

egrep "/usr/bin/rsnapshot monthly: started|/usr/bin/rsnapshot monthly: completed successfully" /var/log/rsnapshot | grep -v "/usr/bin/logger"

7.6 Useful

Reload the systemd daemons

sudo systemctl daemon-reload

List timers

systemctl list-timers

Find rsync processes.

ps -aux | grep "rsync"

7.6.1 Follow a process

Find the process id (PID).

ps -aux | grep rm

Attach to process with strace

sudo strace -s 99 -ffp <PID>

Author: John Herrlin

Created: 2017-04-18 Tue 18:06

Emacs 25.1.1 (Org mode 8.2.10)

Validate