NOTE TO computerconsultantsforum.com and forums.techcareerfubar.com USERS: This is the same site. Your login will work here. Use the "forgot password" function if you need help recovering your password.

Obvious fact: You're not logged in.

Therefore, you're only seeing the tip of the iceberg of great discussion threads on this site.

Get rid of this big black message box by joining here: http://mature-it.pro/register/

Who We Are: A collection of IT, engineering and sciences professionals, in a variety of current circumstances with a variety of career backgrounds. Including System admins, Developers and programmers, Freelancers and "gig" entrepreneurs, Contract, job shopping and FTE-employed contract house IT workers, Web developers, Inventors, and artists and writers with tech backgrounds.

We're smarter than the hive mind you've experienced on large tech discussion forums and groups. So register on the board - your email is NEVER sold or provided to third parties. Then LOGIN FREQUENTLY to see new stuff daily.

Join by Registering here: http://mature-it.pro/register/

« previous next »
Pages: [1]   Go Down

Author Topic: Linux backup: What's now working for me  (Read 359 times)

G0ddard B0lt

  • I absolutely DESPISE improvised sulfur-charcoal-salt peter cannons made out of hollow tree branches filled with diamonds as projectiles.
  • Trusted Member
  • Wise Sage
  • ******
  • Posts: 22672
  • Gorn Classic, user of Gornix
Linux backup: What's now working for me
« on: April 21, 2018, 05:17:25 am »
Here is what is currently working for me, which I have been using since early March several times:

- The file compression utility package "dar." Dar is included in the package manager for Linux Mint. https://en.wikipedia.org/wiki/Dar_%28disk_archiver%29 for more info.

- The backup utility backup-manager. See https://linux.die.net/man/8/backup-manager for details. Also seems to be part of the Mint package set.

DAR has HUGE advantages over all other backup formats. It saves file attributes, and the data storage format is designed to be recoverable even in the event of errors in the archives.

Here is my backup regimen:

Manual, using a portable hard drive. It could be easily automated if I were continuously connected to backup media.

I have created a series of content-specific config files. If you run backup-manager without any arguments it uses a default /etc based config file. Otherwise you can have per-task configs that you pass it on the command line.

I have chopped up backup into about 5 major groups based on total file sizes. The initial master backup of each group took a couple of hours apiece, at least.

- All "business" files: home based directories for email, Quicken/Quickbooks data, website archives, client work directories.
- All "self created media" files: images I take, video clips, etc.
- All downloaded media such as TV programs and movies we watch through streaming
- All Virtualbox hard disk files.
- A backup of /home/gorn but only files NOT covered by above backup groups.

Here's the relevant, important config file statements from the backup-*.conf files hat support my backup style.

# The mounted backup hard drive. Never changes.
export BM_REPOSITORY_ROOT="/media/gorn/SeagateBackup/linuxdesktopbackup"

#This forces incremental backup plus DAR archives (DAR captures all Linux file attributes like ownership, etc
#so I can use any external media such as vfat, SSH based remote drives, etc) Tar does not support many features of DAR.
export BM_ARCHIVE_METHOD="tarball-incremental"

# I left encryption off!

# Example of inclusion of files for a specific backup cluster
export BM_TARBALL_DIRECTORIES="/home/gorn/html /home/gorn/clients /home/gorn/accountingdata /home/gorn/personal /home/gorn/swprojects /home/gorn/agent-email

# For the /home/gorn catchall backup ONLY, I must exclude all of the file paths specified by other backup config files.
# So the following exclusion  statement in the /home/gorn file will look like this:

export BM_TARBALL_BLACKLIST="/home/gorn/images /home/gorn/music /home/gorn/html ...

There is no order in which the various backup configs may be applied. You may also backup some sets of files more frequently than others.

Last night, having an established set of backups already on the external hard drive, I had backup times for each script measuring from a couple of minutes, to 20 minutes for one, to an hour and a half for the 12 GB of changed data for the virtual machine files.

Logged
Gornix is protected by the GPL. *

* Gorn Public License. Duplication by inferior sentient species prohibited.

G0ddard B0lt

  • I absolutely DESPISE improvised sulfur-charcoal-salt peter cannons made out of hollow tree branches filled with diamonds as projectiles.
  • Trusted Member
  • Wise Sage
  • ******
  • Posts: 22672
  • Gorn Classic, user of Gornix
DAR archive handling, after backup - DarGUI
« Reply #1 on: April 21, 2018, 05:31:39 am »
Actually, I already documented the bash script I used to generate these config files:

http://forums.techcareerfubar.com/programming-and-technology/linux-backup-by-timeshift/msg96645/#msg96645

To add to this info:

There is a non-distribution "wrapper" program for DAR called "DarGUI" that makes it easy to work with the generated backups. It is similar in concept to WinZIP on Windows.

I think I got it here:

http://dargui.sourceforge.net/

One VERY important thing I found about these archives. Once you start a set of archives under a particular master name with a particular backup-manager config file, pointing at ANY of the Dar archives related to that master name on the external hard drive will allow you to see ALL of the backup history of those sets of files.

DAR files are extremely "smart."

In other words, extracting backed up data is not fussy at all.

I use DarGUI because the Dar command like switches are fairly difficult to understand and put together, and DarGUI has that knowledge built in.

You can extract/recover your files to any arbitrary path, not just file system root, so it's possible to pull out backups manually for comparison or manual recovery work.

You can, of course, just recover everything onto a new file system.
Logged
Gornix is protected by the GPL. *

* Gorn Public License. Duplication by inferior sentient species prohibited.

benali72

  • CCF Winner's Circle - Supporter
  • Wise Sage
  • *
  • Posts: 2731
Re: Linux backup: What's now working for me
« Reply #2 on: April 21, 2018, 07:33:51 pm »
Thanks so much for sharing the results of your backup quest! This is really useful stuff I'm saving.

It's funny, I think I heard of DAR years ago but forgot all about it. It sounds like a good tool for the job.

I don't know which backup tool is shipped as default in my particular version of Mint. It's labeled BACKUP TOOL in the menu and doesn't have an About box in the product! The execuable is called mintbackup.

Thanks again.
Logged

G0ddard B0lt

  • I absolutely DESPISE improvised sulfur-charcoal-salt peter cannons made out of hollow tree branches filled with diamonds as projectiles.
  • Trusted Member
  • Wise Sage
  • ******
  • Posts: 22672
  • Gorn Classic, user of Gornix
Re: Linux backup: What's now working for me
« Reply #3 on: April 21, 2018, 07:46:42 pm »
At this point what I have set up almost has an Acronis feel to it - flexible, powerful, pretty fast, and a HELL of a lot more reliable than Acronis. But without the commercial lock-in aspects. And allegedly more robustness so that a disk error won't trash an entire archive.

The specific package is called: "backup-manager". The one on my box is version 0.7.10.1-2.

dar on my system is v. 2.5.3-1ubuntu1.

backup-manager will use other archivers other than DAR as a back end, but DAR has the best  capability for creating incremental, compressed archives out to any medium.

backup-manager also can do what the rsync-based backup tools do and just copy and sync files out to a backup drive, but you must use a Unix compatible file system on the storage device in order to do that. dar will create incremental backups on anything, even FAT32.

I also customized the "slice size" which is the size of individual created files. Dar will span the slices as need be if, say you have a huge virtual disk file. In the config file:

# With the "dar" filetype, you can choose a maximum slice limit.
export BM_TARBALL_SLICESIZE="4500M"

About the size of a DVD. Which I have found is a nice size to work with in case I need to copy backups around.
Logged
Gornix is protected by the GPL. *

* Gorn Public License. Duplication by inferior sentient species prohibited.

G0ddard B0lt

  • I absolutely DESPISE improvised sulfur-charcoal-salt peter cannons made out of hollow tree branches filled with diamonds as projectiles.
  • Trusted Member
  • Wise Sage
  • ******
  • Posts: 22672
  • Gorn Classic, user of Gornix
Re: Linux backup: What's now working for me
« Reply #4 on: April 21, 2018, 07:49:21 pm »
I think most of the backup tools built for Linux like mintbackup use rsync as a back end.

Rsync generally means that the tool just does file copies. Is not incremental. And Rsync crashes my box.

I love what I'm now using because the output is so general-purpose. You could FTP/SSH the slice files out to a remote server, no problem.
Logged
Gornix is protected by the GPL. *

* Gorn Public License. Duplication by inferior sentient species prohibited.

benali72

  • CCF Winner's Circle - Supporter
  • Wise Sage
  • *
  • Posts: 2731
Re: Linux backup: What's now working for me
« Reply #5 on: April 25, 2018, 08:36:34 am »
Quote from: The Gorn on April 21, 2018, 07:49:21 pm
I think most of the backup tools built for Linux like mintbackup use rsync as a back end.

Yeah, that's what it looks like. And it's surprising given how you've created a much more flexible backup system using competing tools. Go figure...

Maybe your next step is to sell this. Or contribute it for free to the open source world in exchange for reputation and a useful trump card on your resume.
Logged

G0ddard B0lt

  • I absolutely DESPISE improvised sulfur-charcoal-salt peter cannons made out of hollow tree branches filled with diamonds as projectiles.
  • Trusted Member
  • Wise Sage
  • ******
  • Posts: 22672
  • Gorn Classic, user of Gornix
Re: Linux backup: What's now working for me
« Reply #6 on: April 25, 2018, 09:01:57 am »
The pivot of this mini-project was the package backup-manager. I just figured out a lightweight way to automate the creation of multiple task-specific backup config files.

You must still know a lot to use the generated configs correctly. It's anything BUT a product.

Rsync and straight file copying is a non-starter for backups unless you have total control over the destination backup media and you can format it as a Linux compatible file system. That's what many Linux backup solutions do. Otherwise, you need to use tar (or the more advanced tar replacement dar) if you want to write your backups to anything available.
Logged
Gornix is protected by the GPL. *

* Gorn Public License. Duplication by inferior sentient species prohibited.
Pages: [1]   Go Up
« previous next »