CCF Archive Site

After experimenting with several highly recommended apps for Linux backup, I threw in the towel. They are all too invasive, and they are all uncontrollable.

I have found one that *appears* to be a great alternative.

As a background to this, my methodology in the event of an entirely crashed system  - HDD crash, etc. - will be to recreate my setup using a virgin OS install, and *then* restore my own personal and business data in my accustomed places to the rebuilt machine. I'm not trusting a comprehensive backup and restore solution any more. They all take too much time and I can hone in on the data that I need backed up.

Here's my main beefs: most Linux backup tools are based on rsync to move the data to the mass storage, and most backup programs want to create cron jobs which run whenever the external drive they are configured with is plugged in. So you can't control them or shut them down easily (or at all), and a kill command can shut down your box.

I don't want either. Rsync has issues: I've found repeatedly on my system that if I kill or abort a backup process running rsync, it takes my otherwise completely stable Linux desktop down. It just shuts down and reboots. Rsync is a risk. And these backups (such as fwbackup, backintime, simpleback) rely on rsync.

Also, some of these tools are not hard drive formatting agnostic. fwbackup just copies files to the hard drive with no compression or archiving. Fine, except I have formatted my drive in vfat format, which does not have Unix style permission or owner flags in the file system, so the files are stored with vanilla attributes.

Not acceptable.

I just set up a backup regimen for myself based on the following tool: "backup-manager". I am furthermore using "dar", a highly refined replacement for tar, as the file compression method. Dar saves all Unix attributes, even to a foreign non Unix drive, and it compresses data. Also it supports incremental backups.

I constructed a tool that creates a series of config files that I use to backup several subsets of my user data:

All business (client files, all emails, website archives, saved articles.)
Part of my media: images going back to the late 90s and also family scans
Part of my media: videos, commercial as well as personal
All Virtualbox VMS
The rest of the "/home/Gorn" path apart from the specific groups of backed up files.

The thing is, I want to be able to JUST back up my email+financial data. Without backing up my movies or the changes to the VMs. I want 30 min - 1 hr backups of my critical data, not 5 hour backups that drag in every stinking thing I've done with the PC in the last two months.

This approach lets me do just that.

I just ran through backing up all of my data today with this new approach. It's lengthy initially. Hopefully, incremental backups will be much faster.

If you like I'll share links to the tools and my config file and config generator.

It's not a matter of "not LIKING" rsync.  rsync almost always crashes my PC when I kill its process and most of these backup programs are just wrappers around rsync.

And I don't have a misconfigured system. I can repeat this pattern of behavior. Otherwise my system is very stable.

Basically my goal here was an Acronis like backup of my personal data - compressed, incremental, and able to tune it. Unlike Acronis black box commercial shitware  I don't expect a Linux backup to be easily corrupted and useless as a result.

The config generator wasn't that much work. I had a specific methodology I just described that I followed and doing the string replacements by hand was possible but it would be quite error prone. Honestly, the hardest part was remembering that the sed script commands can use a character other than forward slash / for the search delimiter. Since I used path names with / all over the place for substitutions I used a pipe | character in place of / for sed commands.

Resources for the tools I mentioned:

https://github.com/sukria/Backup-Manager
http://xmodulo.com/linux-backup-manager.html

DAR "replacement" for tar archive utility:

http://dar.linux.free.fr/

Dar itself is an interesting utility. It was engineered as a better "tar". It is alleged in its documentation that an individual file in the dar archive may be corrupt but dar will uncompress everything it can and work around any such irrecoverable errors.

The combination of backup-manager plus dar (I had to install both using apt-get but they were both in the standard distro packages) allows you to do the following:

Compressed, incremental, optionally encrypted backups of the actual file state, including Unix permission and user flags, to any destination device with any file system.

dar is the only compression method supported by backup-manager that does this.

Here's my scripts.

The prototype .conf config file for backup-manager, with wild cards for string replacements. It is a modification of the standard conf file included with the program:

http://posteddocs.nfshost.com/backup-base.conf

The four strings to be replaced for your particular backup are: (assumes a locally connected portable hard drive). Each string looks like #tobereplaced#.

# Where to store the archives
export BM_REPOSITORY_ROOT="#backupdestpath#"

# Prefix of every archive on that box (default is HOSTNAME)
export BM_ARCHIVE_PREFIX="$HOSTNAME"."#backupname#"

# Paths without spaces in their name:
export BM_TARBALL_DIRECTORIES="#backupdirlist#"

# Files to exclude when generating tarballs, you can put absolute
# or relative paths, Bash wildcards are possible.
export BM_TARBALL_BLACKLIST="#backupexcludelist#  /media /home/gorn/backups /home/gorn/Dropbox /home/gorn/mystuff  .thumbnails* .cache* rash* *.backup* *~ .dropbox* /proc/* /sys/*"

So these four wildcard parameters are replaced in the generated config files.

The scripts that generate the config files using the above config file prototype as the source. This contains example paths and data. Replace for your own system and situation:

media1='/home/gorn/images /home/gorn/music '
business='/home/gorn/html /home/gorn/clients /home/gorn/accountingdata /home/gorn/personal /home/gorn/swprojects /home/gorn/agent-email /home/gorn/business /home/gorn/Desktop /home/gorn/websites '
media2='/home/gorn/Videos /home/gorn/movies '
virtualmachines='/home/gorn/VirtualBox_VMs'
backupdestpath='/media/gorn/SeagateBackup/linuxdesktopbackup'

# Parameters
# 1 - Space delimited list of Paths to back 1
# 2 - Directories to exclude
# 3 - name to assign to backup set on the backup media
# 4 - destination path for local backup media (a external HDD)
# 5 - source backup file with wildcard symbols placed
# Output will be configuration file text. Redirect to config file of choice.

./build-backupconf.sh  "$media1"            "" "MediaStuff1" "$backupdestpath" backup-base.conf >backup-mediastuff1.conf
./build-backupconf.sh  "$media2"            "" "mediastuff2" "$backupdestpath" backup-base.conf >backup-mediastuff2.conf
./build-backupconf.sh  "$business"          "" "business"    "$backupdestpath" backup-base.conf >backup-business.conf
./build-backupconf.sh  "$virtualmachines"   "" "vmachines"   "$backupdestpath" backup-base.conf >backup-vmachines.conf

./build-backupconf.sh  "/home/gorn"    "$media1$business$media2$virtualmachines" "homegorn"  "$backupdestpath" backup-base.conf >backup-homegorn.conf

build-backupconf.sh (called as sort of procedure from above):

# Parameters
# 1 - Space delimited list of Paths to back 1
# 2 - Directories to exclude
# 3 - name to assign to backup set on the backup media
# 4 - destination path for local backup media (a external HDD)
# 5 - source backup file with wildcard symbols placed
# Output will be configuration file text. Redirect to config file of choice.

sed -e "s|#backupdirlist#|$1|" -e "s|#backupexcludelist#|$2|"  -e "s|#backupdestpath#|$4|" -e "s|#backupname#|$3|" $5

So this file generates these config files when run:

backup-homegorn.conf 
backup-mediastuff1.conf 
backup-business.conf 
backup-mediastuff2.conf 
backup-vmachines.conf

The idea of the five config scripts is this:

Each one is a related area of backup and is custom to my PC.  If I have a lot of activity with clients, taxes or business, I will apply just the backup-business.conf file to capture the last few days' work.

Each config file when used for backup generates a master backup data set of size up to 120 GB.  Most are smaller. I'm clustering sets of some directories into backups for the reason that I wanted a small set of individual backups, each of which would not be unwieldy to perform size-wise.

For instance one of the largest backup sets I created is for my "images" (all personal photos, few videos) directory which is about 100 GB. It gets its own backup which is backup-mediastuff1.conf.

I attempted doing larger backups of my system and the backup would crash near the end when it was performing an md5 checksum. I found the sizes of the home folders by performing a shell command I'll give below.

The commands that generate the actual backups are performed from a root shell. They look like this (replace the conf with the one germane to your backup slice):

date ; backup-manager --no-upload --no-burn --no-purge -c backup-vmachines.conf ; date

I bracketed the backup command with date commands so I'd get times of start and end, just so I know what to expect next time.

Lastly, note that backup-homegorn.conf is the backup of ALL home user directories that are *not* involved in the specific clusters of backups. They are the stuff left after business, media, etc have been backed up. This includes all of the Linux config files that are stored in the home folder. Note that the generator script's parameter for exclusions for the home directory backup is all of the pre-defined groups of backups. So the home/user backup covers everything not covered by one of the file folder specific backups.

The shell command that reports each /home/gorn  folder size total is as follows. The results from this command will allow you to see how large each of your folders are so that you can customize your own backup scripts using my method. Just go to the home directory you want to assess and run this command line:

find -maxdepth 1 -type d -name '[!.]*' -exec du -sh {} +
You'll get a report that looks like this (example):

...
16K ./temp
115M ./articles
14G ./clients
2.4G ./accountingdata
4.3M ./Desktop
4.5G ./personal
21G ./websites
575M ./gps
406M ./Videos
3.8G ./Downloads
4.0K ./Public
2.4M ./filedatestransfer
1.5G ./swprojects
...


Very nice work, Gorn. Thanks for sharing this! I really appreciate the way you've segregated your backups into manageable-sized pieces.

Sounds like the linux community desperately needs a replacement for Rsync. (I see alternatives out there but I assume they're not even as good as rsync since almost all the developers seem to use rsync as the underlying tool in their higher-level backup packages.)

Cheers.

Thanks Benali72.

I neglected to add (but if one just looks at the code it's apparent) that all of the backup configs created are for non-overlapping sets of files. I think that was probably obvious but I wanted to affirm that.

I'm thinking that there may be a market need for a small e-book of Linux / Gnu recipes for end users who want to dig in. At least I will try to blog this approach.

About rsync, it's considered an extremely standard Gnu utility. I use rsync from cron jobs to move backup files of websites (one huge file consisting of the website's file space + a MySql dump of the database) to an FTP service I use called Adrive.com. It's problem free in that context.

I've described and characterized the issue I have with it. Maybe there is a specific way a background rsync process must be killed.