Easy Automated Snapshot-Style Backups with Linux and Rsync(mikerubel.org)
mikerubel.org
Easy Automated Snapshot-Style Backups with Linux and Rsync
http://www.mikerubel.org/computers/rsync_snapshots
4 comments
http://www.dirvish.org/ and http://rsnapshot.org/ are both wrappers around rsync that do exactly this. Eventually, though, I switched to http://duplicity.nongnu.org/ for the network backends and the encryption.
How do you like duplicity? That looks like a good solution for my work backups.
I haven't pushed it very hard. I'm only using it for personal stuff.
That said, it's pretty good at DWIM for a command-line tool. It works as advertised and doesn't require any fiddling once you have it set up.
cons: command line syntax is verbose. Enough so that there's a wrapper for it (http://duply.net/) which I also recommend. Access to backups isn't as straightforward as with forest-of-hardlinks approaches.
I'd be interested to hear how it holds up to production use.
That said, it's pretty good at DWIM for a command-line tool. It works as advertised and doesn't require any fiddling once you have it set up.
cons: command line syntax is verbose. Enough so that there's a wrapper for it (http://duply.net/) which I also recommend. Access to backups isn't as straightforward as with forest-of-hardlinks approaches.
I'd be interested to hear how it holds up to production use.
some of the comments about rdiff-backup (below) might also apply to duplicity, since it seems to share much of the same internals (duplicity seems to be the encrypted version of rdiff-backup).
Are there any pros/cons to this approach versus rdiff-backup?: http://www.nongnu.org/rdiff-backup/
I use rsnapshot and rdiff-backup, the big difference is rdiff-backup chokes on larger file volumes. I have a backup with a million+ video and image files, at around 600 gigs. rdiff-backup doesn't like this very much, and spends way more time backing it up than rsnapshot. There were other more serious problems, but I made the switch a year or two ago and I can't remember what they were.
Additionally, the braindead simplicity of rsnapshot makes it much easier to verify backups without having to do a restore first.
That said, I use rdiff-backup to back up /etc on all my systems to a local backup directory for quick access if anything gets accidentally fucked up while reconfiguring a service. It really is better for small datasets.
My backup hierarchy, in terms of complexity / size of the job, generally goes: rdiff-backup (small stuff) -> rsnapshot (bigger stuff) -> bacula (mammoth stuff)
Additionally, the braindead simplicity of rsnapshot makes it much easier to verify backups without having to do a restore first.
That said, I use rdiff-backup to back up /etc on all my systems to a local backup directory for quick access if anything gets accidentally fucked up while reconfiguring a service. It really is better for small datasets.
My backup hierarchy, in terms of complexity / size of the job, generally goes: rdiff-backup (small stuff) -> rsnapshot (bigger stuff) -> bacula (mammoth stuff)
I use rdiff-backup too, and since it stores the diffs, you can go back in time.
I also backup the rdiff-backup tree itself to a remote location.
I also backup the rdiff-backup tree itself to a remote location.
rdiff-backup does (more or less) the same thing, but is simpler to use, as far as i can tell (i switched from rsync to rdiff-backup about a year ago, and haven't looked back).
Obvious, but worth pointing out anyway: This is only an option if you trust the remote system you're using for storing your backups and you're sure that it's secure.
Rsyncrypto (http://rsyncrypto.lingnu.com/index.php/Home_Page) offers a partial solution to the "is the remote side secure?" problem (it weakens the cryptosystem used to allow rsync to mostly transfer changes).
As for trust, rsync.net is well established, is run by and for people like us, and seems to do a good job.
As for trust, rsync.net is well established, is run by and for people like us, and seems to do a good job.
a partial solution to the "is the remote side secure?" problem
No. Not at all. Reusing CBC IVs is not a solution to anything.
No. Not at all. Reusing CBC IVs is not a solution to anything.
The point of reusing the IV is that identical blocks of data encrypt identically. That's the information that is leaked. That's the behaviour required by rsync to efficiently transfer changes.
Is it really so bad?
Is it really so bad?
Depending on your attack model, yes, it really can be that bad.
Much better to do your deduplication on the plaintext.
Much better to do your deduplication on the plaintext.
But this is not trying to address a duplication problem as far as I can see (see below):
E.g. let's say I have the following blocks of data:
A B C D E F...
And I then change C to C', now having:
A B C' D E F...
With the rsynccrypto approach the ciphertext of D E and F... will be unchanged, allowing rsync to do the right thing (not retransmit everything after C').
Yeah, this leaks info, but it's a compromise between a making the result easier to break and conserving transmission costs (which is very much an issue for me, being stuck behind a slow upstream residential ADSL link).
My question, which I have most definately not answered to my complete satisfaction, is "has the cryptosystem been compromised to the point I might as well have not bothered?"
Just how easy will the know leaks make a attack feasible? How expensive would be such an attack be? I.e. I'm not concerned about the NSA, a lesser degree of security is OK.
As for deduplication, I don't see how it can help; could you be more specific?
(And thanks a lot for taking this issue seriously, it is something I'm concerned about and don't have a sufficient background to be all that confident about my judgment.)
E.g. let's say I have the following blocks of data:
A B C D E F...
And I then change C to C', now having:
A B C' D E F...
With the rsynccrypto approach the ciphertext of D E and F... will be unchanged, allowing rsync to do the right thing (not retransmit everything after C').
Yeah, this leaks info, but it's a compromise between a making the result easier to break and conserving transmission costs (which is very much an issue for me, being stuck behind a slow upstream residential ADSL link).
My question, which I have most definately not answered to my complete satisfaction, is "has the cryptosystem been compromised to the point I might as well have not bothered?"
Just how easy will the know leaks make a attack feasible? How expensive would be such an attack be? I.e. I'm not concerned about the NSA, a lesser degree of security is OK.
As for deduplication, I don't see how it can help; could you be more specific?
(And thanks a lot for taking this issue seriously, it is something I'm concerned about and don't have a sufficient background to be all that confident about my judgment.)
Or if security isn't a (top) priority.
Regarding making the backups read-only after they are made, but still allowing new backups to be made:
Why not use the snapshot facility of LVM?
Once you are done with a backup, create a snapshot and make it read-only. You can always mount the snapshot to see your data at that point in time and you dont need to worry about the data being erased in the live partition.
Why not use the snapshot facility of LVM?
Once you are done with a backup, create a snapshot and make it read-only. You can always mount the snapshot to see your data at that point in time and you dont need to worry about the data being erased in the live partition.