Ask HN: Any DIY/installable alternatives to Dropbox?
I really like Dropbox (http://www.getdropbox.com) and use it on a daily basis. However, I already have readily accessible capacity on my web server and would rather have the option to run the system on my own. In addition I have occasional clients who require verified encryption of their data. Are there any projects out there that create a similar solution or is this uncharted territory?
26 comments
I run a combination of rsync+git as cron jobs. They keep all 4 computers in the house in sync with each other as well as backups on a NAS.
Why even bother with rsync you may ask? Because for huge binary files (like RAW/JPEG images) I don't need a history of changes and they don't compress well, i.e. git would lead to an increased disk usage.
BTW using "any VCS" advice won't work here. Git can automatically pick up rename/delete/create actions on whole directory trees without explicit commands, which is what you want if you're going for 100% automatic operation: if I delete a file on laptop "A" it will automatically disappear on laptop "B" too. Perhaps git isn't unique in this, but its better among other VCS systems I'm familiar with. It also is very compact, especially if you don't want to have a full-blown uncompressed working tree on a server.
I love Dropbox, but I don't see how I could use it with my 55K/second upload speed, which is, unfortunately, quite typical for US users. (my server is in the closet). If I were them I would seriously consider selling a NAS+software kit. It's hard to imagine a middle class multi-laptop family that doesn't need one of those.
Why even bother with rsync you may ask? Because for huge binary files (like RAW/JPEG images) I don't need a history of changes and they don't compress well, i.e. git would lead to an increased disk usage.
BTW using "any VCS" advice won't work here. Git can automatically pick up rename/delete/create actions on whole directory trees without explicit commands, which is what you want if you're going for 100% automatic operation: if I delete a file on laptop "A" it will automatically disappear on laptop "B" too. Perhaps git isn't unique in this, but its better among other VCS systems I'm familiar with. It also is very compact, especially if you don't want to have a full-blown uncompressed working tree on a server.
I love Dropbox, but I don't see how I could use it with my 55K/second upload speed, which is, unfortunately, quite typical for US users. (my server is in the closet). If I were them I would seriously consider selling a NAS+software kit. It's hard to imagine a middle class multi-laptop family that doesn't need one of those.
Hi Gregg, I like your setup, would you mind describing it a bit more? like a mini tutorial or schema drawing, so it could reproduced easily by another geek like me :)
Or I'll take it as a challenge, and as soon as I've got some time I'll try to do something kind-of like your setup ;)
Or I'll take it as a challenge, and as soon as I've got some time I'll try to do something kind-of like your setup ;)
* Have a linux/osx box that will be your server. Have ssh and git installed and configured.
* Configure all client laptops with passwordless ssh logins to the server (i.e. using certificates) and install git on them.
* Create a bare git repo on the server. I have a repo for each user.
* Have a ~/backup.sh script (better call it backup.command on macs) each of your client laptops that looks something like this (this one is for OSX):
http://pastie.org/358427
Then create a cron job that runs this file as often as you want. It's actually pretty efficient, in my case runs in about 3 seconds if there are no significant changes.
My laptops run it every 20 minutes and my home directory (and music, photos and other stuff) is 100% identical on all computers I work on.
The server, in turn, can be configured to encrypt and push your stuff at night over to your slicehost account or something, when you're not using your connection and this the speed isn't much of an issue.
* Configure all client laptops with passwordless ssh logins to the server (i.e. using certificates) and install git on them.
* Create a bare git repo on the server. I have a repo for each user.
* Have a ~/backup.sh script (better call it backup.command on macs) each of your client laptops that looks something like this (this one is for OSX):
http://pastie.org/358427
Then create a cron job that runs this file as often as you want. It's actually pretty efficient, in my case runs in about 3 seconds if there are no significant changes.
My laptops run it every 20 minutes and my home directory (and music, photos and other stuff) is 100% identical on all computers I work on.
The server, in turn, can be configured to encrypt and push your stuff at night over to your slicehost account or something, when you're not using your connection and this the speed isn't much of an issue.
"Configure all client laptops with passwordless ssh logins to the server (i.e. using certificates) "
Short version of this step:
on each client, run ssh-copy-id user@server
Short version of this step:
on each client, run ssh-copy-id user@server
You could hack something together using rsync, or unison. But, frankly, if neither of these things does what you want out of the box you will quickly find yourself spending more time than it's worth to save the $99 annual cost of a 50GB Dropbox account.
I use unison to sync a bunch of folders between my laptop and desktop, but only on an ad hoc basis. For stuff that I want synced in the background I use Dropbox. And, frankly, I'm not sure why I'm not using Dropbox for everything. For example, I'm running Jungledisk to make backups of my current project folders to S3 in the event of a big fire in my building... but if I just put my Jungledisk stuff in a Dropbox I believe I'd be syncing it and backing it up with versioning, all automatically. I need to look into that.
As for the clients that require verified encryption... is it possible to make an encrypted volume, created with something like Truecrype, and stick that in Dropbox? I've done that. Not sure how the task of automating the mounting and dismounting would go, nor of how secure this really is.
I use unison to sync a bunch of folders between my laptop and desktop, but only on an ad hoc basis. For stuff that I want synced in the background I use Dropbox. And, frankly, I'm not sure why I'm not using Dropbox for everything. For example, I'm running Jungledisk to make backups of my current project folders to S3 in the event of a big fire in my building... but if I just put my Jungledisk stuff in a Dropbox I believe I'd be syncing it and backing it up with versioning, all automatically. I need to look into that.
As for the clients that require verified encryption... is it possible to make an encrypted volume, created with something like Truecrype, and stick that in Dropbox? I've done that. Not sure how the task of automating the mounting and dismounting would go, nor of how secure this really is.
Here's an article that reviews two possibly relevent open source tools - http://www.linux.com/feature/154149
Unison File Synchronizer: http://www.cis.upenn.edu/~bcpierce/unison
DirSync Pro: http://directorysync.sourceforge.net/
Unison File Synchronizer: http://www.cis.upenn.edu/~bcpierce/unison
DirSync Pro: http://directorysync.sourceforge.net/
BackupPC http://backuppc.sourceforge.net/ is probably what you're looking for, although I can't tell you about the encryption part. I am just starting to look into serious backup solutions myself.
I have set up BackupPC before. It is not quite the same as dropbox and probably won't fill the client file-sharing and change tracking need; it is a backup system, that you can retrieve files and directories via a browser. It is more like SonicWall's CDP or something (it is not continuous and on-the-fly like SonicWall's product, the last I checked, but with the newer linux kernels having inotify built in, it is technically possible to do it).
[deleted]
I used Novell iFolder for a while (running on Ubuntu, though it was a pain to install). I think it worked pretty well on SuSE out of the box. Anyway, it uses a client on individual machines with a server hosting the files. Pretty much like Dropbox, though I think Dropbox is a little prettier and easier to use.
http://www.novell.com/products/ifolder/overview.html
https://help.ubuntu.com/community/iFolderClient
http://www.novell.com/products/ifolder/overview.html
https://help.ubuntu.com/community/iFolderClient
Yes it would be potentially useful/lucrative to be able to license and rebrand the Dropbox technology (server + clients) but self-hosted.
[Edit: no, not the Dropbox technolgy; the Dropbox experience.]
[Edit: no, not the Dropbox technolgy; the Dropbox experience.]
Hmm, having a private repo might be pretty nice.
I wonder if the Dropbox cats have thought about doing something like this? They could continue acting as a "controller" and route data elsewhere.
I wonder if the Dropbox cats have thought about doing something like this? They could continue acting as a "controller" and route data elsewhere.
For an rsync-like solution that also handles the tar-ing and (GnuPG) encryption of whatever directory you want to backup, take a look at duplicity (if you're on some Linux flavor). http://duplicity.nongnu.org/
It is a command-line solution, and simple to throw in cron. In addition, you can use it to back things up to S3 directly. It's great.
It is a command-line solution, and simple to throw in cron. In addition, you can use it to back things up to S3 directly. It's great.
It seems to me that the file-sharing and version tracking are two different things that should have two different problems.
If you want this but on your own servers, maybe check out ringlight: http://ringlight.us/
If you want this but on your own servers, maybe check out ringlight: http://ringlight.us/
Slightly off topic here, but as many people are mentioning rsynch, I was wondering if it is possible to get rsynch working on a cellphone/smartphone.
I was intrigued by the idea of making a universal backup/synch client based on rsynch.
I was intrigued by the idea of making a universal backup/synch client based on rsynch.
I always thought that dropbox would be much better if I could tie it together with my S3 account and just have all the data saved there, to an unlimited amount paid by me.
hmmm... i've been looking for something like this for a while now also.
I've also considered making one of my own.
Webdav + some simple scripting could do it on any server. That's basically what the "iDisk" for .Mac is.
One of the really great aspects of Dropbox is that it versions all content and just transmits/saved the delta's after the original file is saved. It there an easy way of achieving that around a webdav share?
Just use any vcs. afaik there is svn behind Dropbox.
im really interested in this, also.
SVN with webdav and apache would work.
Or roll that together with incron/rsync for a more "in the background" approach.
unfortunately it has an CJK characters issue.