Clone Dropbox with a Raspberry Pi and BTSync(reustle.io)
reustle.io
Clone Dropbox with a Raspberry Pi and BTSync
http://reustle.io/blog/btsync-pi
61 comments
This is the same issue I have with btsync, so about a month ago I started working on an open-source clone, with similar features but an open, documented protocol. https://github.com/jewel/clearskies
best of luck. i really hope you succeed with this. it seems that right now all opensource dropbox alternatives, while great, have a shortcoming somewhere (git annex, sparkleshare, btsync...)
I'm looking forward to clearskies and will gladly switch over to it once it's ready!
though BTsync is "meant for large files" I wouldn't really advise syncing truecrypt files as it syncs the entire file not just updating lines in the file... (though that would probably be much more efficient) even still using truecrypt would still require entire file syncs the second anything changed. :/ maybe instead encrypt each file separately using AES... might be slightly better as you won't be syncing whole gigs of files if you only changed a small text file...
Or encfs, a per file encrypted filesystem.
I'm using BTSync with EncFS, but if you don't trust the BTSync binary – it's probably got access to the non-encrypted data as well unless you've jumped through a few more hoops than a "typical BTSync installation". I'd guess 99.9% of BTSync users have the binary installed with at least enough permissions to read anything the logged in user can read (which includes the cleartext-mounted stuff in your EncFS or TrueCrypt volumes).
On top of that (literally and figuratively) is Pi's ARM's TrustZone hypervisor..
[deleted]
I installed and played with BTSync yesterday. It was really easy to get it all up and running syncing between two local machines and a remote server. The config was really simple compared to other stuff I've used. Sometimes it could take a while before the syncing would start which is a bit of a pain.
We run a system where we have a lot of data on a server that we want to see locally. Edits are made both locally and on the server. At the moment we're using Aerofs to handle the syncing. As a general rule it works really well. The issue we've run into is with pre-seeding new machines (ie, when our spot instance based server get killed). I need to play around a bit more but it's time consuming to get a machine back online and totally synced (there may be a solution to this).
Mostly it's a pain being at the mercy of someone else's magic. Aero is great, except when it isn't - and we don't know the rules. That's a real pain if it's business critical. BTSync looked great - until I realised that it's not opensource either.
There are a few syncing options on there but they don't all tick all the boxes. Unison doesn't trigger on fs events so you have to run it manually. BTSync isn't opensource. Dropbox syncs to their servers and back. With AeroFS you don't always know how it's doing what it's doing.
So far AeroFS has been the winner for us. BTSync definitely looks like it fills the same space though.
We run a system where we have a lot of data on a server that we want to see locally. Edits are made both locally and on the server. At the moment we're using Aerofs to handle the syncing. As a general rule it works really well. The issue we've run into is with pre-seeding new machines (ie, when our spot instance based server get killed). I need to play around a bit more but it's time consuming to get a machine back online and totally synced (there may be a solution to this).
Mostly it's a pain being at the mercy of someone else's magic. Aero is great, except when it isn't - and we don't know the rules. That's a real pain if it's business critical. BTSync looked great - until I realised that it's not opensource either.
There are a few syncing options on there but they don't all tick all the boxes. Unison doesn't trigger on fs events so you have to run it manually. BTSync isn't opensource. Dropbox syncs to their servers and back. With AeroFS you don't always know how it's doing what it's doing.
So far AeroFS has been the winner for us. BTSync definitely looks like it fills the same space though.
We tested AeroFS and BTSync on a variety of data in shared folders, and consistently found sync issues with BTSync. We eventually abandoned BTSync for AeroFS, and while it hasn't been perfect, it's been considerably better at handling syncing issues.
Anecdotal, and just one case, but hope someone finds it helpful.
Anecdotal, and just one case, but hope someone finds it helpful.
We have had the same experience. I loved the idea of BTsync, but we encountered too many issues. We tested sparkleshare and lipsync too. Unfortunately, none of them dealt with conflicts and merges as they were meant to.
BTsync would often not pick up changes (using osx and linux clients), and/or overwrite new files with an older version. It also had a slow latency when a file did change. As to the open source nature of it, you can supposedly limit its access to the shared servers through configuration, but this caused more sync problems.
These tests made us appreciate the Dropbox implementation even more.
Unfortunately, we need non-cloud syncing for our data, so I wrote a complex cron/rsync/inotify hack with its own issues - but at least we know the limitations and have custom workarounds. Personally, I hope BTsync will mature and be a feasible alternative.
BTsync would often not pick up changes (using osx and linux clients), and/or overwrite new files with an older version. It also had a slow latency when a file did change. As to the open source nature of it, you can supposedly limit its access to the shared servers through configuration, but this caused more sync problems.
These tests made us appreciate the Dropbox implementation even more.
Unfortunately, we need non-cloud syncing for our data, so I wrote a complex cron/rsync/inotify hack with its own issues - but at least we know the limitations and have custom workarounds. Personally, I hope BTsync will mature and be a feasible alternative.
That's starting to sound like a pattern on BTSync now.
What do you mean by non-cloud syncing? Have you tried Aero? I thought about doing a inotify/unison solution - apparently unison is good at dealing with conflicts and the like.
With all these syncing tools it's the uncertainty that causes issues. You're always worried about what data is going to end up where in the edge cases. As you say - at least with a few scripts glued together you know what's happening (and how to fix it if required).
What do you mean by non-cloud syncing? Have you tried Aero? I thought about doing a inotify/unison solution - apparently unison is good at dealing with conflicts and the like.
With all these syncing tools it's the uncertainty that causes issues. You're always worried about what data is going to end up where in the edge cases. As you say - at least with a few scripts glued together you know what's happening (and how to fix it if required).
Non-cloud = my clumsy way of saying operating on private servers with no 3rd party storage (and ideally no connection) necessary.
Looked into Aero, but I am pushing towards an open source alternative to be used, or at least where the implementation is fully controlled/configured by us.
I agree, unison is great and I think is the way forward towards a private dropbox solution.
Looked into Aero, but I am pushing towards an open source alternative to be used, or at least where the implementation is fully controlled/configured by us.
I agree, unison is great and I think is the way forward towards a private dropbox solution.
Useful to know. I only tested btsync briefly yesterday and it wasn't picking up changes as quick as Aero. Again, single data point but I'd recommend Aero from what I've seen (and we use it all day every day).
Does AeroFS support 32bit OSen yet? That was a showstopper for me back when it first became available - it _required_ 64 bit Java…
I have done something similar, but used a Raspberry Pi and Seafile (http://seafile.com) instead of BTSync. I did not get the chance to test it with large media files, but works great for documents and photos. Also has mobile and cross platform clients.
And the best part is it's open-source: https://github.com/haiwen/seafile
And the best part is it's open-source: https://github.com/haiwen/seafile
I'm using Seafile with a VPS. I had the VPS anyway for other projects and it's great to be able to access my files from anywhere. I'm really impressed with Seafile's polish, too.
The only feature I miss is not being able to host files publicly so others can download them like Dropbox lets you. I ended up with a workable solution though by installing a seafile client on the seafile server as well. I then used it to sync a particular directory with my PC and made that directory available through nginx.
The only feature I miss is not being able to host files publicly so others can download them like Dropbox lets you. I ended up with a workable solution though by installing a seafile client on the seafile server as well. I then used it to sync a particular directory with my PC and made that directory available through nginx.
You can generate shared link to a file or folder on your seafile website and send it to others.
Could you share your experiences, for example how many files do you have on the server? I have read[0] that Seafile struggles with larger amounts of files and I was hoping to use it as a drop-in NAS replacement for businesses.
[0] http://blog.patshead.com/2013/06/self-hosted-cloud-storage-c...
[0] http://blog.patshead.com/2013/06/self-hosted-cloud-storage-c...
It doesn't need to be a USB 3.0 drive. The RPi is USB 2.0 only.
I've stopped reading when he really recommended that you get a USB 3.0 drive to connect to a RPi :P
I kept reading expecting an explanation for this, but there isn't one. Does he understand the difference? Who knows.
The explanation of "every step of the build" is pretty limited too.
The explanation of "every step of the build" is pretty limited too.
I do understand this but what is wrong with recommending picking up a USB 3 external drive over USB 2? I doubt the Pi is the only device it will be used with.
"I am still working on the best way to access the data without disconnecting the drive from the Raspberry Pi every time I want to add/remove something."
It sounds like he's putting the data on the USB drives from other computers, not from the RPi.
It sounds like he's putting the data on the USB drives from other computers, not from the RPi.
with a 150Mbps wifi adapter
plus, the RPi can't even achieve full USB 2.0 speed, it is not a good choice for a file server.
FWIW, in this use-case the limit is the wifi or more likey your internet connection speed. You don't really need blazing disk read/write speeds for your "Dropbox clone" - if you can read and write data as fast as your adsl connection goes - anything more is "wasted". As a general purpose file server though - yeah, a 'Pi isn't going to stream multiple HD videos to all 4 bedrooms on your home network.
When I setup a file-server for myself, I set it up on the local network I use most, so the Internet speed is relevant, but not that much.
Both my wifi and internet connection speed are well over the 2megabytes/s I'm getting on my pi, which really is the only downside I have found with my pi.
btsync is really nice, just install the software, no account creation needed, and if you just want to sync computers on a LAN, no internet connection needed. I was worried about btsync being closed source but when I looked at alternatives like dropbox and spideroak I noticed they weren't open either and you have to sign up for an account and give them personal information to get serious amounts of storage.
Everyone who's looking to run their own private Dropbox should check out ownCloud[1]. It's free, open source, you install it on your own server (or Raspberry Pi), and it's not P2P like BTsync is. It works more like Dropbox, with a centralized server. You just run that server yourself.
They have apps for Windows, Mac, Linux, iOS and Android. I've been using it for about two months now with no problems.
[1] http://owncloud.org
They have apps for Windows, Mac, Linux, iOS and Android. I've been using it for about two months now with no problems.
[1] http://owncloud.org
OwnCloud is quite nice, but its Dropbox-like feature isn't his best asset. We have big shared folders and the client app does weird stuff sometimes (and given the way things are architected, I'm not confident it will change).
On the other hand, it gives you a nice web-based file browser, CardDav, CalDav, and a RSS webapp. And there's a basic iPhone app.
The approach posted somewhere else in the comments[0] seems interesting. Use BTSync for the synchronization, and put ownCloud over that for the web interface and other goodies. I'm tempted to experiment with that.
[0] http://blog.bittorrent.com/2013/05/23/how-i-created-my-own-p...
On the other hand, it gives you a nice web-based file browser, CardDav, CalDav, and a RSS webapp. And there's a basic iPhone app.
The approach posted somewhere else in the comments[0] seems interesting. Use BTSync for the synchronization, and put ownCloud over that for the web interface and other goodies. I'm tempted to experiment with that.
[0] http://blog.bittorrent.com/2013/05/23/how-i-created-my-own-p...
I used ownCloud for both the CardDav/CalDav/etc. and the dropbox-like stuff for a couple of months now. Sadly as you say the dropbox functionality doesn't work well. It takes up to several days until every file is synchronized everywhere and I want to synchronize almost my whole /home directory between my ThinkPad and my iMac.
A couple of days ago I installed btsync and tested syncing one 130 MB big directory. I added it both at the same time to btsync and ownCloud. After about 2 minutes btsync had its copy on the other computer. After one hour ownCloud had cpied about 50 MB to the server when I canceled it and removed the ownCloud dropbox-client from my computers.
I still use all the other stuff to synchronize my calendar/address book between my computers and mobile devises and I even use their RSS backend but with my own Firefox OS app https://marketplace.firefox.com/app/feedmonkey/
A couple of days ago I installed btsync and tested syncing one 130 MB big directory. I added it both at the same time to btsync and ownCloud. After about 2 minutes btsync had its copy on the other computer. After one hour ownCloud had cpied about 50 MB to the server when I canceled it and removed the ownCloud dropbox-client from my computers.
I still use all the other stuff to synchronize my calendar/address book between my computers and mobile devises and I even use their RSS backend but with my own Firefox OS app https://marketplace.firefox.com/app/feedmonkey/
The CardDav and CalDav stuff was a hassle for me to make work. After the trial and error and googling, getting it set up myself, I pretty much had to grab each family member's mobile device or computer and punch in the magic codes. But once I had it going... It works! All of our calendar and contact data is available on each of our computers and we can share them easily.
And we own the data, not Apple.
I highly recommend owncloud.
And we own the data, not Apple.
I highly recommend owncloud.
Then there are git-annex assistant (No Windows version) and SparkleShare, which are both Open Source and should be able to do the same.
Missing from both though is a mobile version. I quite like my Elements (A text editor for Dropbox files).
Missing from both though is a mobile version. I quite like my Elements (A text editor for Dropbox files).
Even if it's nice, I won't recommend it to anyone. If it becomes big, I don't want the next Edward Snowden to tell me it's been used by .gov in the future.
No but the way Dropbox and btsync work are way way different.
> BitTorrent Sync synchronizes your files using a peer-to-peer (P2P) protocol.
> BitTorrent Sync synchronizes your files using a peer-to-peer (P2P) protocol.
They are different, but Dropbox does have some "P2P" behavior with LAN sync:
https://www.dropbox.com/help/137/en
Note: Dropbox employee, but this is public information.
https://www.dropbox.com/help/137/en
Note: Dropbox employee, but this is public information.
How well does BTSync work in terms of a large team collaborating together? Dropbox has a habit of just working, with surprisingly few hiccups (none so far except during massive folder migrations). Does BTSync work that well with a team of 10? 20? 30? that work on the same set of documents (though not on the same document simultaneously, so no need for e.g. git).
Given that it's using the Bittorrent protocol (Basically) to handle the actual transfers, yes it will work; the issue you may run into is that there can be a lot of files and data available in the share and every client will try to download all files.
The challenge here is not data transfer, but interacting with the underlying filesystem well -- handling, say, one user saving a file while another is moving it into a subdirectory, or programs that save files by deleting and rewriting them (or writing a temporary file and then moving it), or similar. That's the real strength of Dropbox.
That happens with Dropbox too though.
Actually not. Dropbox has a feature called Selective Sync which lets you choose which files are synced to which computers.
We're not using that. Dropbox is a shared drive for the company.
My question, still unanswered, remains whether anyone has made btsync work well in a setup where it's used as a shared drive for 10-20 people who interact with many files, though usually not the same file simultaneously (though that happens occasionally, and Dropbox seems to support it...)
My question, still unanswered, remains whether anyone has made btsync work well in a setup where it's used as a shared drive for 10-20 people who interact with many files, though usually not the same file simultaneously (though that happens occasionally, and Dropbox seems to support it...)
Based on neumann comment above, btsyng still has a lot of issues.
Keep using Dropbox.
Keep using Dropbox.
Hi, I'm doing something similar,
but instead of BTsync, I use ssh & unison/rsync.
The remote pi calls back the main pi by ssh. The main pi uses this connection to sync files. So the remote pi can be behind a firewall and no issue.
Im interesting in sharing the setup - and improving it, if anyone interested let me know. Im at padrone.nl
The remote pi calls back the main pi by ssh. The main pi uses this connection to sync files. So the remote pi can be behind a firewall and no issue.
Im interesting in sharing the setup - and improving it, if anyone interested let me know. Im at padrone.nl
>unison
The forgotten gem.
It was a poster child for OCaml. Didn't get the attention it deserved.
The forgotten gem.
It was a poster child for OCaml. Didn't get the attention it deserved.
I used my dropbox for really important files. But then all the rest is archives and don't mind it getting it next day. All my external hard drives never last more than 2 years. While it is an interesting approach I am concerned about reliability if it is an archive.
I tried out Bitcasa for Mac and it is terrible. My Finder basically stop responding after bitcasa started. I have been working on a simplified version of AWS boto library for glacier, in the goal of making an open source tool like a command line sftp/ftp style access to glacier.
I tried out Bitcasa for Mac and it is terrible. My Finder basically stop responding after bitcasa started. I have been working on a simplified version of AWS boto library for glacier, in the goal of making an open source tool like a command line sftp/ftp style access to glacier.
It is the umpteenth time I see a "Dropbox clone" on HN, but none of them is.
I badly want an open-source sync online storage client and server, cross-platform, with encryption at least om a server side, that I can install on my own VPS or my own host at home.
Online storage became an important element of a digital life, pretty much like Web earlier. I know that Dropbox is an YC company, but I wonder where are Apaches and Nginxes of the online storage?
I badly want an open-source sync online storage client and server, cross-platform, with encryption at least om a server side, that I can install on my own VPS or my own host at home.
Online storage became an important element of a digital life, pretty much like Web earlier. I know that Dropbox is an YC company, but I wonder where are Apaches and Nginxes of the online storage?
I think Seafile checks all your boxes. I run it on a VPS and use the Windows, Linux and Android clients daily without any issues. I haven't tried the encryption but it allows you to password-protect folders.
http://seafile.com/en/home/
http://seafile.com/en/home/
See jewel's comment below https://news.ycombinator.com/item?id=6648109
I really liked the implementation listed out on this site. I went through it and it was a great learning experience. You also get a sweet web interface with owncloud.
http://blog.bittorrent.com/2013/05/23/how-i-created-my-own-p...
http://blog.bittorrent.com/2013/05/23/how-i-created-my-own-p...
My friend ran BTSync on a raspi, and it was dreadfully slow to index for large amounts of data. Like, multiple days of 100% CPU slow.
Also, with encrypted volumes, just reading files on the raspi is very slow. Consider replacing raspi with odroid, utilite, or other fast, fanless, cheap arm sbc.
If the 'Pi is just a "cloud storage appliance", there's no need for it to care that the data is encrypted.
I use EncFS to encrypt on my laptops/desktops/servers/phones/tablets, and have BTSync sync the encrypted partitions to a machine at home that doesn't have EncFS installed. I'm using an old Mac Mini rather than a 'Pi, but the same principle works - the Mac Mini serves my archive/backup needs for my synced data (it's using software raid1 over two drives for reliability, and does TimeMachine backups to a rotating set of external drives for archiving), but none of the data on the Mini or it's drives is of any use if hacked/stolen, since if doesn't have EncFS or the crypto keys.
I use EncFS to encrypt on my laptops/desktops/servers/phones/tablets, and have BTSync sync the encrypted partitions to a machine at home that doesn't have EncFS installed. I'm using an old Mac Mini rather than a 'Pi, but the same principle works - the Mac Mini serves my archive/backup needs for my synced data (it's using software raid1 over two drives for reliability, and does TimeMachine backups to a rotating set of external drives for archiving), but none of the data on the Mini or it's drives is of any use if hacked/stolen, since if doesn't have EncFS or the crypto keys.
I've been wanting to cancel Dropbox as well. But I choose a different path which includes Bitcasa. I wrote a blogpost about it earlier: http://blog.noort.be/2013/08/02/backup-in-2013.html
I tried this recently... it works well except with the mobile integration. I then purchased a TonidoPlug and things seemed to go much more smoothly.
I wouldn't use it sync anything except Truecrypt volumes. :)