Ask HN: how do you backup your servers?
I need to think about backing up data I have on a server, and would like to know how you do it. Do you backup whole machines or just some specific directories? Do you do off-site backups, or local? How much data do you have in a backup job? What's you preferred method: tar, sftp, scp, rsync, paid-for services etc? What's the most annoying thing you encounter managing backups?
28 comments
I divide my backups into three types:
So I have a bare metal backup of a compressed image of my system drive (made with dd using a live-CD) which is updated every time I mess with LVM and nightly snapshots of my root partition (LVM snapshop of partition with dd).
From that toe-hold I can rebuild my system from any worse case short of hardware destruction, I use nightly backups with BackupPC for normal system recovery, augmented with what's below for frequently changed data.
Frequently changing data, like email and code I'm working on, is rdiff-backup snapshotted every hour. If I was maintaining customer data I'd do a continuous process.
The same data is rsyncsrypto-ed and most of that is rynced to rsync.net.
Occasionally changing data is rsynced every night.
I try to avoid common mode failures, e.g. no one piece of software, one system or one disk will cause total data loss.
Ask if you want particular details; for a long treatment of this, read Backup & Recovery, "Inexpensive Backup Solutions for Open Systems" by W. Curtis Preston (http://www.amazon.com/Backup-Recovery-Inexpensive-Solutions-...). It's getting to be a bit old, but a lot of it still holds and the principles will be true for a long time.
(Note, while the above particularly paranoid approach is for my personal systems, it's based on harsh experience going back to 1977 (sic), including plenty of small scale "enterprise" systems where total data loss was unacceptable and a couple of 24x7 Internet startups.)
Bare metal.
Frequently changing data.
Occasionally changing data.
So I have a bare metal backup of a compressed image of my system drive (made with dd using a live-CD) which is updated every time I mess with LVM and nightly snapshots of my root partition (LVM snapshop of partition with dd).
From that toe-hold I can rebuild my system from any worse case short of hardware destruction, I use nightly backups with BackupPC for normal system recovery, augmented with what's below for frequently changed data.
Frequently changing data, like email and code I'm working on, is rdiff-backup snapshotted every hour. If I was maintaining customer data I'd do a continuous process.
The same data is rsyncsrypto-ed and most of that is rynced to rsync.net.
Occasionally changing data is rsynced every night.
I try to avoid common mode failures, e.g. no one piece of software, one system or one disk will cause total data loss.
Ask if you want particular details; for a long treatment of this, read Backup & Recovery, "Inexpensive Backup Solutions for Open Systems" by W. Curtis Preston (http://www.amazon.com/Backup-Recovery-Inexpensive-Solutions-...). It's getting to be a bit old, but a lot of it still holds and the principles will be true for a long time.
(Note, while the above particularly paranoid approach is for my personal systems, it's based on harsh experience going back to 1977 (sic), including plenty of small scale "enterprise" systems where total data loss was unacceptable and a couple of 24x7 Internet startups.)
Cron + S3 + s3cmd + small custom script for DB dumps
Cron + snapshots + small custom script for EBS on EC2.
Basically, I store all non code files on EBS and redundantly backup to S3, which amounts to living or dying by Amazon's hand.
Recovery consists of firing up a new EC2 instance, attaching a snaphot and running a deploy script, which is handled automatically by a different EC2 instance, assuming I haven't lost all instances. Manual recovery in that case would be time consuming, but doable.
Snapshots are nearly instantaneous and uploads to S3 from an EC2 instance are about as fast as you can get.
Most annoying thing? Testing recovery.
Cron + snapshots + small custom script for EBS on EC2.
Basically, I store all non code files on EBS and redundantly backup to S3, which amounts to living or dying by Amazon's hand.
Recovery consists of firing up a new EC2 instance, attaching a snaphot and running a deploy script, which is handled automatically by a different EC2 instance, assuming I haven't lost all instances. Manual recovery in that case would be time consuming, but doable.
Snapshots are nearly instantaneous and uploads to S3 from an EC2 instance are about as fast as you can get.
Most annoying thing? Testing recovery.
"Most annoying thing? Testing recovery."
Indeed ... but at least you do that!
One of the great things about clouds like AWS is that you can now do an almost complete bare metal restore test (modulo stuff like your official DNS vs. test) for a small amount of money instead of trying to justify buying a separate system for that sort of thing (which in the real world I was never able to do, although I fortunately never suffered a total data loss despite a few close calls).
Indeed ... but at least you do that!
One of the great things about clouds like AWS is that you can now do an almost complete bare metal restore test (modulo stuff like your official DNS vs. test) for a small amount of money instead of trying to justify buying a separate system for that sort of thing (which in the real world I was never able to do, although I fortunately never suffered a total data loss despite a few close calls).
I backup entire disks using a towers of hanoi algorithm with (wait for it...) good old dump(8) to a second local hard drive. This method keeps 7 full days of backups (using an incremental diff, so it's fairly space efficient). I've used this method for almost 15 years and have used it dozens of times to recover missing things. dump has never failed me!
On my other servers I use rsync just to keep a current mirror around. I always ask myself, "How hard would this be to recover if I lost it?" and "Will I need to go back in time for some things?" Depending on the answer to those questions I make my backup strategy for the server in question.
Yesterday's Daring Fireball post on the same topic said it best: "Every hard drive in the world will eventually fail. Assume that yours are all on the cusp of failure at all times. It’s good to be spooked about how long your hard drives will last."
On my other servers I use rsync just to keep a current mirror around. I always ask myself, "How hard would this be to recover if I lost it?" and "Will I need to go back in time for some things?" Depending on the answer to those questions I make my backup strategy for the server in question.
Yesterday's Daring Fireball post on the same topic said it best: "Every hard drive in the world will eventually fail. Assume that yours are all on the cusp of failure at all times. It’s good to be spooked about how long your hard drives will last."
I've used xfsdump/xfsrestore on Linux and xfs-filesystems (obviously) in the past. Very nice stuff, I second the recommendation.
Of course it only provides for the raw data-transfer task so one has to think about the scheduling and planning yourself.
Of course it only provides for the raw data-transfer task so one has to think about the scheduling and planning yourself.
dump(8) on Linux?
An acquaintance with decades of experience in the hard disk industry including a fist full of patents including 2-3 in every disk drive you own says:
"One should start planning the death of the drive, and all one's data, beginning the day one buys the drive."
An acquaintance with decades of experience in the hard disk industry including a fist full of patents including 2-3 in every disk drive you own says:
"One should start planning the death of the drive, and all one's data, beginning the day one buys the drive."
Generally rsync to a mirror-image hardware box in the same location. Everything is backed up so any fault/failure on the primary machine can be recovered (in theory) via the hot-standby machine.
Critical user data, databases, etc. are dump'd/tar'd/gzip'd to a remote location once a day, generally held as long as space permits.
Critical user data, databases, etc. are dump'd/tar'd/gzip'd to a remote location once a day, generally held as long as space permits.
[deleted]
Considering the work we do (corporate and criminal computer forensics) backup strategies are about our second most important thing :D So naturally it gets neglected...
Gold Images: of the various OS configurations and requisite software so you can redeploy a server fairly quickly
Diff Backups taken nightly of our various data stores / databases
Weekly backups of main server data (MySQL, MSSQL, Hadoop clusters etc)
In terms of software we use a couple of commercial solutions plus Bacula (which isn't really that great unless your using tapes...).
Inspired by Tarsnap (the encryption and so forth) Im in the process of speccing up an idea to write out own in house version (I'd love to use Tarsnap directly but not a chance unfortunately) to replace our mish mash of software.
Gold Images: of the various OS configurations and requisite software so you can redeploy a server fairly quickly
Diff Backups taken nightly of our various data stores / databases
Weekly backups of main server data (MySQL, MSSQL, Hadoop clusters etc)
In terms of software we use a couple of commercial solutions plus Bacula (which isn't really that great unless your using tapes...).
Inspired by Tarsnap (the encryption and so forth) Im in the process of speccing up an idea to write out own in house version (I'd love to use Tarsnap directly but not a chance unfortunately) to replace our mish mash of software.
tarsnap + bash script! http://www.tarsnap.com/
Tar-like interface, data deduplication, awesome support, fair pricing. What more could you ask for?
Tar-like interface, data deduplication, awesome support, fair pricing. What more could you ask for?
We also use tarsnap, would certainly recommend
Nice. I'm using S3 right now, but since Tarsnap is tuned for backups, I'm gonna check it out.
I'm just a mortal developer so I don't have to admin anything serious, but here's how I do backups of my personal files & server. Mainly using Git:
Any code / changes to Debian's default /etc setup go into Git and are at least on both the relevant machine (laptop or server) and GitHub. See http://github.com/avar and http://github.com/avar/linode-etc for the /etc on the server. I keep some personal data (like finance notes) in GPG encrypted files in private Git repositories.
E-Mail is backed up my Google's GMail (presumably) and data like my small collection of music is something I can always get back from the Internet.
If the filesystem on my laptop, server or GitHub is destroyed right now I can get everything working again relatively quickly just by installing Ubuntu or Debian from scratch and copying my home directory, /etc and code from Git on one of the remaining machines.
I've made a script that automatically backs up my Git repositories daily (http://github.com/avar/github-backup), the resulting backup is around 180MB. Losing some of my other data would be an annoyance but this is all I /really/ care about.
Follow somebody else's advice if you need "real" backups, but for my simple needs this suffices.
Any code / changes to Debian's default /etc setup go into Git and are at least on both the relevant machine (laptop or server) and GitHub. See http://github.com/avar and http://github.com/avar/linode-etc for the /etc on the server. I keep some personal data (like finance notes) in GPG encrypted files in private Git repositories.
E-Mail is backed up my Google's GMail (presumably) and data like my small collection of music is something I can always get back from the Internet.
If the filesystem on my laptop, server or GitHub is destroyed right now I can get everything working again relatively quickly just by installing Ubuntu or Debian from scratch and copying my home directory, /etc and code from Git on one of the remaining machines.
I've made a script that automatically backs up my Git repositories daily (http://github.com/avar/github-backup), the resulting backup is around 180MB. Losing some of my other data would be an annoyance but this is all I /really/ care about.
Follow somebody else's advice if you need "real" backups, but for my simple needs this suffices.
We've been using Bacula: weekly fulls and daily incrementals to disk, and weekly fulls to tape for offsite storage. As our data size grows, tapes are going to run out of steam. I've recently began experimenting with http://www.nongnu.org/duplicity to do encrypted backups to S3. So far it has been working great.
We do tar'ed directories/databases onto S3 and this script: http://simonwillison.net/static/2008/backup_to_s3.py.txt
Right now its only done weekly, as we're not experiencing high growth. We also store downloads, so the tars can get very big, but it works well.
Right now its only done weekly, as we're not experiencing high growth. We also store downloads, so the tars can get very big, but it works well.
I backup code to flash once a day in case of hardware failure. Once a week I swap flash drives into a safety deposit box. I also backup any local databases to flash once a day. Use a scheduled task and a .cmd for that, which automatically stops SQL Server, 7z compresses and encrypts the database files and source code repository, then move them to a usb flash drive, then restarts SQL Server.
I never upload code to the internet, but I have copies of the repository in multiple locations separated by at least 3000 miles.
Nightly I backup internet databases to a cloud storage provider. Nondatabase files are stored simultaneously on local storage and cloud storage. The local storage is mirrored to local snapshots hourly and nightly in case a roll back is needed.
I never upload code to the internet, but I have copies of the repository in multiple locations separated by at least 3000 miles.
Nightly I backup internet databases to a cloud storage provider. Nondatabase files are stored simultaneously on local storage and cloud storage. The local storage is mirrored to local snapshots hourly and nightly in case a roll back is needed.
I don't bother backing up the full system, just the user generated data. I am storing lots of data as flatfiles rather than shoving them into the database, so rsync helps there.
I like having a complete copy of all user-data on my home machine for development purposes as well as backup. So, I rsync all user generated content (they are 99% flatfiles so this is easy), as well as a daily mysql dump. My other founder and I both perform this daily, so at any point we are only exposed with a 1 day risk (seems acceptable at this point). This also gives us two separate off-site backups, which is nice.
I like having a complete copy of all user-data on my home machine for development purposes as well as backup. So, I rsync all user generated content (they are 99% flatfiles so this is easy), as well as a daily mysql dump. My other founder and I both perform this daily, so at any point we are only exposed with a 1 day risk (seems acceptable at this point). This also gives us two separate off-site backups, which is nice.
an openbsd vm in a... secret location... runs rsnapshot (rsync over ssh) to all of my servers every few hours and keeps separate hourly, daily, weekly, monthly, and yearly "full" backups. they are stored on a usb hard drive encrypted with openbsd's softraid.
when the server boots up, i have to enter in the softraid passphrase, it mounts the drive, then i launch screen and start up the script that spawns the rsnapshot processes at the appropriate times. when starting it, i enter in the common ssh key passphrase (which is restricted on all the servers to only allow logging in from that network) and common mysql backup user password. these are stored in memory and then the script runs rsnapshot every few hours.
once a day, the script does a full mysql dump of all databases on all servers using a ruby script with net::ssh to tunnel mysql access over ssh. it does this to keep "proper" mysql backups in case the rsync'd binary mysql files are transferred in an odd state, and keep daily backups of everything. it also lets me quickly get at data with standard text tools instead of having to restore the raw mysql files somewhere temporarily, start up mysqld, and extract it.
so once the server is booted, it's completely automated. i've been doing backups this way for a couple years and have recovered files and databases from them a number of times. no complaints.
when the server boots up, i have to enter in the softraid passphrase, it mounts the drive, then i launch screen and start up the script that spawns the rsnapshot processes at the appropriate times. when starting it, i enter in the common ssh key passphrase (which is restricted on all the servers to only allow logging in from that network) and common mysql backup user password. these are stored in memory and then the script runs rsnapshot every few hours.
once a day, the script does a full mysql dump of all databases on all servers using a ruby script with net::ssh to tunnel mysql access over ssh. it does this to keep "proper" mysql backups in case the rsync'd binary mysql files are transferred in an odd state, and keep daily backups of everything. it also lets me quickly get at data with standard text tools instead of having to restore the raw mysql files somewhere temporarily, start up mysqld, and extract it.
so once the server is booted, it's completely automated. i've been doing backups this way for a couple years and have recovered files and databases from them a number of times. no complaints.
I have scripts (I use fabric, keep meaning to explore chef) that bring a fresh OS install up to a production server automatically, doing all installation and configuration. This is all in a DVCS system, so I have a local copy of my repository as well as a hosted copy of my repository on bitbucket/github. That takes care of code and configuration.
I have a cron job that dumps the databases into a directory with a time-stamped name. The last few regular nightly back-ups are kept, then weekly for a couple of weeks, then monthly for about a year. The parent directory is then backed up to S3 using duplicity. I've done other, cleverer things in the past with rotating through buckets based on the month, keeping every backup, doing them twice a day, that sort of thing.
I test recovery by spinning up another server, running the scripts to bring it up to production status, and running the script that uses duplicity to pull the data dump directory back out of S3, and then restores the database from the dump.
At $0.15 a gig to store data, and about $0.10 a gig to transfer it in, the cost of S3 is pretty much just noise. I've got on-disk dumps of my database if the database is corrupted on my machine, and I've got back-ups of those dumps if I need them.
Duplicity is pretty fantastic -- definitely check it out.
I have a cron job that dumps the databases into a directory with a time-stamped name. The last few regular nightly back-ups are kept, then weekly for a couple of weeks, then monthly for about a year. The parent directory is then backed up to S3 using duplicity. I've done other, cleverer things in the past with rotating through buckets based on the month, keeping every backup, doing them twice a day, that sort of thing.
I test recovery by spinning up another server, running the scripts to bring it up to production status, and running the script that uses duplicity to pull the data dump directory back out of S3, and then restores the database from the dump.
At $0.15 a gig to store data, and about $0.10 a gig to transfer it in, the cost of S3 is pretty much just noise. I've got on-disk dumps of my database if the database is corrupted on my machine, and I've got back-ups of those dumps if I need them.
Duplicity is pretty fantastic -- definitely check it out.
duplicity is awesome. can backup to gmail IMAPS, $0.00 per gig for storage and $0.00 per gig for transfer in a high end GOOG datacenter. ;)
No whole-system backups. Offsite (across the country) replicated MySQL database with daily full snapshots and periodic encrypted dumps to S3. Real-time encrypted mogilefs backups to S3 on a per-file basis. S3 MFA + versioning is a huge win in both cases. rsync for a small number of exceptions: IMAP data, development trees, etc. When a server fails, just re-image it, bootstrap with puppet, start it back up.
I do a lot of simple things. Servers: use cron, zip, and s3cmd tools to backup to S3; on EC2, use snapshots. Development laptops: use git repos hosted on a rented VPS and I have a cron script to ZIP these repos and back them up to S3.
The data lives separate from the code: the code is backed up in Subversion, the data lives in AmazonS3 (no other backup) for images, and a database in an Amazon snapshot with daily backups automated.
Rsync.net, simple, cheap reliable, and helpful. If you call their telephone # you get a real person, that knows WTF is going on, not some L1 support tech.
EC2 AMI image -> S3 at milestones, mysql dbase backup to S3 every few minutes, all files stored directly on S3, so only cached on server.
duplicity is great for off-site backups - full encryption, supports a wide variety of destinations (amazon s3, FTP etc)
We use ZRM to automate the backup and versioning of our MySQL databases.
Then we use cron and s3sync to mirror the relevant portions of the filesystem to S3.
For those interested, the process is documented here:
http://monk.ly/bFBd7w
Then we use cron and s3sync to mirror the relevant portions of the filesystem to S3.
For those interested, the process is documented here:
http://monk.ly/bFBd7w
[deleted]
Daily rsync to BQBackup, with once weekly offline backup.
Cron + rsync.net and their geo-replicated package.
[deleted]
ZFS auto snapshots, backup-server running incremental ssh "zfs send" |zfs recv.
r1soft
[deleted]