What To Back Up?

[Michael L Torrie]

Kimball Larsen wrote:
> I have heard (and shared) differing opinions about what bits of a  
> linux box really should be backed up in order to resurrect a system  
> after something like catastrophic hard drive failure.
> 
> I am curious what the consensus is here.

It all depends on  your specific needs.  At work I have many servers
that each server specific functions.  Thus I rarely back up full servers
to our off-site backup.

To help out, each server's special configuration is stored in
subversion.  For a mail server, I subversion all the configuration files
that make it run like:

sendmail
dovecot
cyrus sasl
clamav
mimedefang
custom milters
global procmail settings
file server mounts (autofs config)
mailman config
maintenance scripts in /usr/local/bin
etc

As I install the server I commit to subversion an non-trivial
configuration I make.  I typically don't worry about things like ldap
and kerberos client config, since that's always the same on each
machine.  I check out the config files in place in the file system.
Basically the svn tree mirrors the real file system.  On some machines I
subversion things in /etc, /usr/libexec (custom scripts),
/usr/local/libexec, /usr/local/bin, /var/stuff, etc.  This has saved our
bacon on numerous occasions when we destroyed a config file during
editing, or completely hosed a DNS zone with a few misplaced keystrokes
in vim (followed by :wq ;).

My nightly backup to removable disk that goes off-site is never a full
backup either.  The same stuff that's in svn plus the actual data.

However, I do do a full (rsync -ravH) backup of each of my virtual
servers to another Xen host.  I write some fancy scripts to do a full
hot backup, one file at a time.  This will get us back up and running in
very short order.

Non virtual servers have their entire hard drive rsync'ed to a second
internal drive.  A failure in the primary drive would take me as long as
it takes to swap the drives and fix grub to get running again.   No 5
nine's around here, but maybe 3.

Even without the full, everything backups, I can rebuild any one server
in about an hour, provided it's the same exact OS, with the subversion
stuff mainly, then restore the data from backup disk.

Keeping a list of installed packages that are important to a server's
function is also highly recommended.  I am not good at this and usually
rediscover the dependencies as I'm building the new server.

On my home machine I backup the entire drive (full, using rsync) to a
spare internal disk every so often.  I'm not a RAID-1 believer--most
people seem to use it to solve the wrong problems.  I also periodically
do a full backup to a usb disk, which I then actually use in vmware when
I'm at my folks house and need to do something with a real OS.  When
upgrading the OS, I always make a spare copy of /etc, /opt, /usr/local/,
and /var for good measure.  Although last time I did a reinstall, I
forgot that I had a nice cross compiler in /usr/i386-mingw32 so I lost
that.  I'll rebuild it to /opt next time, as God means me to do.