Difference between revisions of "Rsync backup"

From Noah.org
Jump to navigationJump to search
m
 
(11 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
[[Category:Engineering]]
 
[[Category:Engineering]]
 +
[[Category:Free_Software]]
 +
[[Category:Shell]]
  
This is the small script I use for online backups with rsync.
+
See also [[rsync notes]].
This includes a one week rotating window. Prior days include only
 
the files that changed, so each day does not take up the full
 
amount of disk. The prior days' backups will appear complete because
 
identical files are hard linked.
 
This isn't super fancy, but it gets the job done.
 
  
To use this script add something similar to the following line to /etc/crontab:
+
This is the small script I use for backups with rsync. This includes a one week rotating window. Prior days include only the files that changed, so each day does not take up the full amount of disk. The prior days' backups will appear complete because identical files are hard linked. There's about a hundred versions of this script on the Internet. This one isn't super fancy, but it's small and gets the job done.
    01 4 * * * backup /usr/bin/rsync_backup /media/shareddocs /home/backup
 
  
=== rsync_backup.sh script ===
+
When the script complete successfully it will create a file called BACKUP_TIMESTAMP in the root directory of the destination. If the script fails for any reason it will create a file called BACKUP_ERROR. The files will contain the finish date and exit code and message reported by `rsync`.
<pre>
 
#!/bin/sh
 
#
 
# This maintains a one week rotating backup. This will normalize permissions on
 
# all files and directories on backups. It has happened that someone removed
 
# owner write permissions on some files, thus breaking the backup process. This
 
# prevents that from happening. All this permission changing it tedious, but it
 
# eliminates any doubts. I could have done this with "chmod -R +X", but I wanted
 
# to explicitly set the permission bits.
 
#
 
# Pass two arguments:
 
#    rsync_backup SOURCE_PATH BACKUP_PATH
 
#
 
# Noah Spurrier 2007
 
# $Id$
 
  
usage="usage: rsync_backup [-v] SOURCE_PATH BACKUP_PATH\n\
+
This script can change the permission of all directories to 755 and all files to 644. This is specified with the -n flag. This may be a bad thing in some cases. In my case, I am using this as a backup for a shared NAS through SaMBa. For restore purposes the difference between Windows permissions and UNIX permissions is worthless. Normalizing the permissions makes the the files easier to browse through the UNIX side and it removes the problem where someone might change permissions in the backup target directory which could cause rsync to fail.
    SOURCE_PATH and BACKUP_PATH may be ssh-style remote paths; although,\n\
 
    BACKUP_PATH is usually a local directory where you want the\n\
 
    backup set stored."
 
  
while getopts ":vh" options; do
+
== Setup ==
    case $options in
 
        v ) VERBOSE=1;;
 
        h ) echo -e $usage
 
            exit 1;;
 
        \? ) echo -e $usage
 
            exit 1;;
 
        * ) echo -e $usage
 
            exit 1;;
 
    esac
 
done
 
shift $(($OPTIND - 1))
 
SOURCE_PATH=$1
 
BACKUP_PATH=$2
 
if [[ ( -z $SOURCE_PATH ) || ( -z $BACKUP_PATH ) ]]; then
 
    echo "Missing argument. Give source path and backup path in this format:"
 
    echo "    $0 SOURCE_PATH BACKUP_PATH"
 
    echo -e $usage
 
    exit 1
 
fi
 
SOURCE_BASE=`basename $SOURCE_PATH`
 
PERMS_DIR=755
 
PERMS_FILE=644
 
if [ $VERBOSE ]; then
 
    RSYNC_OPTS="-a --delete -v"
 
    date
 
else
 
    RSYNC_OPTS="-a --delete -q"
 
fi
 
  
# Create the rotation directories if they don't exist.
+
This script doesn't require much setup. It will automatically create destination backup directories if they don't exist. After the first time you run it you will see that it creates numbered copies of the source base directory. Each numbered copy appears to be a separate copy, but due to the use of hardlinks each copy takes no more room than one copy plus whatever files have changed between each run of the backup. If you run the backup script from a daily cron job then each numbered directory will correspond to one day in the past.
if [ ! -d $BACKUP_PATH ] ; then
 
    mkdir $BACKUP_PATH
 
fi
 
if [ ! -d $BACKUP_PATH/$SOURCE_BASE.0 ] ; then
 
    mkdir $BACKUP_PATH/$SOURCE_BASE.0
 
fi
 
if [ ! -d $BACKUP_PATH/$SOURCE_BASE.1 ] ; then
 
    mkdir $BACKUP_PATH/$SOURCE_BASE.1
 
fi
 
if [ ! -d $BACKUP_PATH/$SOURCE_BASE.2 ] ; then
 
    mkdir $BACKUP_PATH/$SOURCE_BASE.2
 
fi
 
if [ ! -d $BACKUP_PATH/$SOURCE_BASE.3 ] ; then
 
    mkdir $BACKUP_PATH/$SOURCE_BASE.3
 
fi
 
if [ ! -d $BACKUP_PATH/$SOURCE_BASE.4 ] ; then
 
    mkdir $BACKUP_PATH/$SOURCE_BASE.4
 
fi
 
if [ ! -d $BACKUP_PATH/$SOURCE_BASE.5 ] ; then
 
    mkdir $BACKUP_PATH/$SOURCE_BASE.5
 
fi
 
if [ ! -d $BACKUP_PATH/$SOURCE_BASE.6 ] ; then
 
    mkdir $BACKUP_PATH/$SOURCE_BASE.6
 
fi
 
  
# TODO All these find operations to clean up permissions is going to add a lot
+
== crontab ==
# of overhead as the backup set gets bigger. At 40GB it's not a big deal. The
 
# correct thing would be to have an exception based system where I correct
 
# permissions when they cause a problem.
 
  
# Rotate backups.
+
To use this script add something similar to the following line to /etc/crontab:
find $BACKUP_PATH/$SOURCE_BASE.6 -type d -exec chmod $PERMS_DIR {} \;
+
 
find $BACKUP_PATH/$SOURCE_BASE.6 -type f -exec chmod $PERMS_FILE {} \;
+
     01 4 * * * backup /usr/bin/rsync_backup /media/shareddocs /home/backup
rm -rf $BACKUP_PATH/$SOURCE_BASE.6
 
mv     $BACKUP_PATH/$SOURCE_BASE.5 $BACKUP_PATH/$SOURCE_BASE.6
 
mv    $BACKUP_PATH/$SOURCE_BASE.4 $BACKUP_PATH/$SOURCE_BASE.5
 
mv    $BACKUP_PATH/$SOURCE_BASE.3 $BACKUP_PATH/$SOURCE_BASE.4
 
mv    $BACKUP_PATH/$SOURCE_BASE.2 $BACKUP_PATH/$SOURCE_BASE.3
 
mv    $BACKUP_PATH/$SOURCE_BASE.1 $BACKUP_PATH/$SOURCE_BASE.2
 
cp -al $BACKUP_PATH/$SOURCE_BASE.0 $BACKUP_PATH/$SOURCE_BASE.1
 
  
# Backup.
+
== rsync_backup script ==
find $BACKUP_PATH/$SOURCE_BASE.0 -type d -exec chmod $PERMS_DIR {} \;
 
find $BACKUP_PATH/$SOURCE_BASE.0 -type f -exec chmod $PERMS_FILE {} \;
 
rsync $RSYNC_OPTS $SOURCE_PATH/. $BACKUP_PATH/$SOURCE_BASE.0/.
 
find $BACKUP_PATH/$SOURCE_BASE.0 -type d -exec chmod $PERMS_DIR {} \;
 
find $BACKUP_PATH/$SOURCE_BASE.0 -type f -exec chmod $PERMS_FILE {} \;
 
  
# Create a timestamp file to show when backup process was complete.
+
Click here to download: [http://www.noah.org/engineering/src/shell/rsync_backup rsync_backup]
date > $BACKUP_PATH/$SOURCE_BASE.0/BACKUP_TIMESTAMP
+
<include src="/home/noahspurrier/noah.org/engineering/src/shell/rsync_backup" highlight="sh" />
  
# Create a shortcut to the latest backup.
+
== Ken's exclude.list file ==
ln -sf $BACKUP_PATH/backup.0 $BACKUP_PATH/latest
 
  
if [ $VERBOSE ]; then
+
My friend, Ken, uses this rsync exclude list.
    date
 
fi
 
</pre>
 
  
=== Ken's exclude.list file ===
 
 
<pre>
 
<pre>
 
### tmp ###
 
### tmp ###

Latest revision as of 17:12, 24 April 2014


See also rsync notes.

This is the small script I use for backups with rsync. This includes a one week rotating window. Prior days include only the files that changed, so each day does not take up the full amount of disk. The prior days' backups will appear complete because identical files are hard linked. There's about a hundred versions of this script on the Internet. This one isn't super fancy, but it's small and gets the job done.

When the script complete successfully it will create a file called BACKUP_TIMESTAMP in the root directory of the destination. If the script fails for any reason it will create a file called BACKUP_ERROR. The files will contain the finish date and exit code and message reported by `rsync`.

This script can change the permission of all directories to 755 and all files to 644. This is specified with the -n flag. This may be a bad thing in some cases. In my case, I am using this as a backup for a shared NAS through SaMBa. For restore purposes the difference between Windows permissions and UNIX permissions is worthless. Normalizing the permissions makes the the files easier to browse through the UNIX side and it removes the problem where someone might change permissions in the backup target directory which could cause rsync to fail.

Setup

This script doesn't require much setup. It will automatically create destination backup directories if they don't exist. After the first time you run it you will see that it creates numbered copies of the source base directory. Each numbered copy appears to be a separate copy, but due to the use of hardlinks each copy takes no more room than one copy plus whatever files have changed between each run of the backup. If you run the backup script from a daily cron job then each numbered directory will correspond to one day in the past.

crontab

To use this script add something similar to the following line to /etc/crontab:

   01 4 * * * backup /usr/bin/rsync_backup /media/shareddocs /home/backup

rsync_backup script

Click here to download: rsync_backup <include src="/home/noahspurrier/noah.org/engineering/src/shell/rsync_backup" highlight="sh" />

Ken's exclude.list file

My friend, Ken, uses this rsync exclude list.

### tmp ###
- **/tmp/**
### transient ###
- /proc/
- /mnt/
- /cdrom/
- /media/
- /sys/
- /var/run/
- /var/lock/
- /var/log/
- /var/tmp/
- /dev/
### config ###
- /etc/adjtime
- /etc/motd
- /etc/mtab
- /etc/resolv.conf
- /etc/lvm/.cache
- /var/lib/alsa/asound.state
- /var/lib/dhcp3/dhclient.*.leases
- /var/lib/urandom/
- /var/lib/urandom/random-seed
### apps ###
# firefox
- **/.mozilla/firefox/*.default/Cache/**
# bash
- .bash_history
# svn
- .svn/
- .svn/**
# vim
- .viminfo
- .*.swp
# X
+ .cache/sessions/
- .cache/sessions/**
- .xsession-errors
- .ICEauthority
- .serverauth.*
# less
- .lesshst
### backup ###
- *~