Dropbox: the new file system of the web(collaborable.com)
collaborable.com
Dropbox: the new file system of the web
http://collaborable.com/blog/dropbox-the-new-file-system-of-the-web
63 comments
You can vote for it here: https://www.dropbox.com/votebox/5972/events-api
I too am hoping for an improvement that will scale to a million users. But here's what I do that should scale to tens of thousands decently:
I run dropboxd on my web server. Instead of using the API, users share a folder with my app. Dropboxd gets the stream of changes, and my app polls the local filesystem for changes instead of polling across the net.
I run dropboxd on my web server. Instead of using the API, users share a folder with my app. Dropboxd gets the stream of changes, and my app polls the local filesystem for changes instead of polling across the net.
Whoa, that's really smart. Going to try that!
Or even setting a callback URL for a file or folder that POSTs some JSON when the file changes. Simpler for everyone.
That's what this means:
"If developers could define WebHooks to receive events, that'd be a great start. I'd happily pay for that."
"If developers could define WebHooks to receive events, that'd be a great start. I'd happily pay for that."
Hm. Never seem that particular term used for it before. My mistake.
Mistakes are how we learn! :)
Below is a good write up by Timothy Fiz [1]. It's old, but it's still a good summary. He references GitHub, which is worth noting. GitHub allows developers to set webhooks for automated tasks that are triggered by repo events. It's a great reference implementation with good documentation [2].
1 - http://timothyfitz.wordpress.com/2009/02/09/what-webhooks-ar...
2 - http://help.github.com/post-receive-hooks/
Below is a good write up by Timothy Fiz [1]. It's old, but it's still a good summary. He references GitHub, which is worth noting. GitHub allows developers to set webhooks for automated tasks that are triggered by repo events. It's a great reference implementation with good documentation [2].
1 - http://timothyfitz.wordpress.com/2009/02/09/what-webhooks-ar...
2 - http://help.github.com/post-receive-hooks/
Parent addressed that: that's what a WebHook is.
I could build a Dropbox notification app in an afternoon or two, and sell it to you and a few thousand friends for $5/mo :)
Eric, ideas are cheap and execution is hard. I happen to believe that you're completely out to lunch if you think you could build this in a practical, scalable way that is useful in the ways I describe above.
However, I'm open minded and you seem confident — so if you believe that you're sitting on something, please feel free to share how you'd structure such a service. If you get it right, I'd pay you a lot more than $5, as long as it had the blessing of Dropbox (ie. doesn't get API throttled and isn't going to get cut after I integrate with your tool).
However, I'm open minded and you seem confident — so if you believe that you're sitting on something, please feel free to share how you'd structure such a service. If you get it right, I'd pay you a lot more than $5, as long as it had the blessing of Dropbox (ie. doesn't get API throttled and isn't going to get cut after I integrate with your tool).
I was being facetious -- I do think a service like this could be built, but I am working on something else. Sorry if it came off offensive (I'm not out to lunch).
Really I meant to point out how the pattern of web apps connected via REST API to desktop folders/files is interesting and will enable a lot of data management schemes that were previously much harder or obscure.
Really I meant to point out how the pattern of web apps connected via REST API to desktop folders/files is interesting and will enable a lot of data management schemes that were previously much harder or obscure.
I really like Dropbox. In fact, it's fair to say I love Dropbox. It's made so many use cases so much simpler. However, I am f*g sick of this perception that Dropbox is the silver bullet for every storage solution. This sort of sensationalism is just stupid.
Dropbox is a cloud-hosted, distributed system and just like any other large, distributed system, it's subject to some very real constraints in terms of latency and reliability.
In other words, it's not fair to claim that Dropbox is some magic silver bullet that will solve every scaling problem you're subject to, act like a rock solid filesystem which behaves identically to the one you have locally, or solve your love life problems.
Maintaining this perception is just going to create frustration for people who expect it to act this way, or worse, create a living hell for the developers maintaining the glue code between your system and the Dropbox API.
You try figuring out what the fuck is going on with your system when the "spooky" or "weird" version regressions start occurring. The arms will raise in a giant, collective shrug. Watch in horror as your http requests timeout because the roundtrip time between your box and Dropbox is that bad. The sodden faces of your product customers and the million "(so-and-so's conflicted copy of " files laying around your filesystem which JUST. KEEP. COMING. BACK. will be enough to make you wish you never drank the kool aid.
Now, I'm not saying that the Dropbox REST API isn't the best thing since sliced bread. All I'm saying is it has it's purposes and there are use cases and times when it simply isn't an appropriate choice.
In short, please respect the fact that distributed storage is a hard problem with hard constraints, stop proselytizing Dropbox as the 2nd coming, and relax and enjoy your backed up media and documents on your iPad already. Seesh.
Dropbox is a cloud-hosted, distributed system and just like any other large, distributed system, it's subject to some very real constraints in terms of latency and reliability.
In other words, it's not fair to claim that Dropbox is some magic silver bullet that will solve every scaling problem you're subject to, act like a rock solid filesystem which behaves identically to the one you have locally, or solve your love life problems.
Maintaining this perception is just going to create frustration for people who expect it to act this way, or worse, create a living hell for the developers maintaining the glue code between your system and the Dropbox API.
You try figuring out what the fuck is going on with your system when the "spooky" or "weird" version regressions start occurring. The arms will raise in a giant, collective shrug. Watch in horror as your http requests timeout because the roundtrip time between your box and Dropbox is that bad. The sodden faces of your product customers and the million "(so-and-so's conflicted copy of " files laying around your filesystem which JUST. KEEP. COMING. BACK. will be enough to make you wish you never drank the kool aid.
Now, I'm not saying that the Dropbox REST API isn't the best thing since sliced bread. All I'm saying is it has it's purposes and there are use cases and times when it simply isn't an appropriate choice.
In short, please respect the fact that distributed storage is a hard problem with hard constraints, stop proselytizing Dropbox as the 2nd coming, and relax and enjoy your backed up media and documents on your iPad already. Seesh.
I'm with you; I've already experienced various spooky things with desktop applications on top of Dropbox (like the excellent Papers2 by mekentosj) because of file conflicts and latency. And that's not Papers2's fault, since it would have to be redesigned to be tolerant of such things. But it goes a long way toward showing that Dropbox is not a syncing FS that can put any local application data in the cloud. In particular, Dropbox is bad for typical database use patterns (simultaneous writes, transactions) and the versioning and locking are not tunable.
You're right in the sense that the pattern is to be excited about, and not only Dropbox itself.
AFAIK, Dropbox is a 'filesystem' that does not support partial file overwrites, does not do locking in a meaningful way, etc.
Because of that, anything built along the lines outlined in this post will be brittle. For example, if one updates that 'products.csv', one must be absolutely sure that one applies those updates to the last version saved. A solid tool would have to detect races and offer merging of the files. I would rather use some SQL in the cloud or, lacking that, some SCM. Those, at least, come with tools to manually/automatically handle such race conditions.
Because of that, anything built along the lines outlined in this post will be brittle. For example, if one updates that 'products.csv', one must be absolutely sure that one applies those updates to the last version saved. A solid tool would have to detect races and offer merging of the files. I would rather use some SQL in the cloud or, lacking that, some SCM. Those, at least, come with tools to manually/automatically handle such race conditions.
To be fair, opportunistic locking is implemented in the API (you can pass a revision ID and the call fails if this is not the current version, or something like that).
That would be optimistic locking.
This seriously disturbs me. Dropbox has several fundamental flaws that make it entirely unsuitable for widespread adoption as anything other than a personal file store for people who don't care about privacy. A "filesystem of the web" would need security, encryption, fine-grained sharing controls, reliability, and numerous other features. Meanwhile, the whole sync mechanism of Dropbox doesn't really matter for a "filesystem of the web", because changes can just occur directly in the remote filesystem rather than locally with syncing.
Now, if someone called S3 "the new filesystem of the web", that would make more sense. Numerous services build on S3, including Dropbox itself, and it provides enough tools and API to allow those services to include all the critical features mentioned above. Add a few features for users to hook up their own S3 data to services rather than the S3 accounts of those services, and you'd have something even more interesting. (The same thing applies to any equivalent service to S3, and many such services exist.)
Now, if someone called S3 "the new filesystem of the web", that would make more sense. Numerous services build on S3, including Dropbox itself, and it provides enough tools and API to allow those services to include all the critical features mentioned above. Add a few features for users to hook up their own S3 data to services rather than the S3 accounts of those services, and you'd have something even more interesting. (The same thing applies to any equivalent service to S3, and many such services exist.)
Meanwhile, the whole sync mechanism of Dropbox doesn't really matter for a "filesystem of the web", because changes can just occur directly in the remote filesystem rather than locally with syncing
Excel can't make changes directly to the remote filesystem. That's the novelty of this scheme. It isn't yet another cloud database, it's a way to share local files with cloud apps.
I have mixed feelings about this.
It could serve to bring data out of the silos, and back under user control. That would be an epic win.
But it's not federated. And I anticipate problems with access control, which needs to be per app, per file, and separate read/write.
Aside: I recently had a nasty scare when a shitty iOS password manager deleted my Keepass database from my Dropbox. If hadn't found a copy in the Dropbox cache on my laptop, I would have been in a world of shit.
Excel can't make changes directly to the remote filesystem. That's the novelty of this scheme. It isn't yet another cloud database, it's a way to share local files with cloud apps.
I have mixed feelings about this.
It could serve to bring data out of the silos, and back under user control. That would be an epic win.
But it's not federated. And I anticipate problems with access control, which needs to be per app, per file, and separate read/write.
Aside: I recently had a nasty scare when a shitty iOS password manager deleted my Keepass database from my Dropbox. If hadn't found a copy in the Dropbox cache on my laptop, I would have been in a world of shit.
I'm writing this reply from a commuter train with wifi that goes away whenever we enter a tunnel. My point of view is that a filesystem of the web has to support disconnected operation. The Coda filesystem has worked that way for decades.
Dropbox has now officially replaced "My Documents" on all my computers, Windows and Linux. I recently reinstalled Windows 7 on my gaming PC. When it finished up, I installed DropBox, logged in to my account and went to watch a movie. A couple hours later, I was synced up. Too simple.
So, yes, I can't wait for more web apps to allow me to save my data to my Dropbox!
So, yes, I can't wait for more web apps to allow me to save my data to my Dropbox!
The security implications (which dropbox do not take seriously) of this are a nightmare. I feel sorry for all of the network admins and security people who have to deal with the fact that all of their employees are unintentionally exfiltrating everything.
I'm building one
OK, at work we use Sharepoint to collaboratively edit and push out data to colleagues as Word and Excel files.
Will your app let me do the same kind of thing at home with outside work friends and the voluntary work I do?
Will your app let me do the same kind of thing at home with outside work friends and the voluntary work I do?
I think Dropbox would already enable the collaborative edit and push thing you described with outside work friends, depending on how you define "collaboratively edit".
On top of that, it should be easy to build a web app to accomplish the same thing, plus some extra elegant features.
The question is, would you pay for it? ;)
On top of that, it should be easy to build a web app to accomplish the same thing, plus some extra elegant features.
The question is, would you pay for it? ;)
Can you check files out with DropBox, so you don't have two people working on a file (since presumably they don't have merge working for Word/Excel docs).
I seriously hate your use case. The locked file is the most troublesome thing that seems to negatively impact everyone using these systems so one person can, once in a while, ensure no conflicts happen. A better solution would be smart merging. Just look at source control to see how half-assed locking is as a solution to this problem.
/end-rant
/end-rant
I think this sentiment is reason enough to always leverage source-code control for something like this. They've been working on the problem for an ultra-long time in computer terms, so adapting one's use cases to the capabilities of existing foundational components is probably the quickest way to long-term happiness in the revisable storage world.
SCC works great for text files. In my Dropbox I ha e Excel, Powerpoint, PSD, Keepass, Installers, PDF, MP4, MP3, etc...
Of you have a smart merger for all of these integrated into Dropbox, send me a link.
Of you have a smart merger for all of these integrated into Dropbox, send me a link.
They don't have merge working for _anything_. They simply bypass it by noting that the same file has changed on two computers at once and saving two copies in the folder, for the users to merge themselves.
Much safer that way.
Much safer that way.
Yep, you could do that rather easily with a web app + Dropbox API.
Have y'all tried Box.net?
Drag and Drop file uploads/downloads in HTML5:
http://tutorialzine.com/2011/09/html5-file-upload-jquery-php...
WebDAV mounts as a network share out of the box in Win/OSX/Lin
I love dropbox but I could see it being much worse than usual for a fair % of users
WebDAV mounts as a network share out of the box in Win/OSX/Lin
I love dropbox but I could see it being much worse than usual for a fair % of users
I wonder if anyone has the ability to understand and use dropbox this way but isn't able to mount network shares. I feel like anyone who would use this would see it as worse than usual. Dropbox and it's clones are great, and it's good to see this convenience available to a more mainstream audience.
But as someone who has used sshfs for years to get seamless access to all my boxes from wherever I am I'm certainly not getting excited about some kind of crippled filesystem-over-http fad. It took mobile to remind people that the internet is good for more than just the web but I don't see that trend stopping (even if we do have to start calling software "native apps")
The web's great, but this everything-in-the-browser silliness means people are using crappier tools than they used to have. I find it annoying but practically it's just not a stable state of affairs because it's so easy to change.
But as someone who has used sshfs for years to get seamless access to all my boxes from wherever I am I'm certainly not getting excited about some kind of crippled filesystem-over-http fad. It took mobile to remind people that the internet is good for more than just the web but I don't see that trend stopping (even if we do have to start calling software "native apps")
The web's great, but this everything-in-the-browser silliness means people are using crappier tools than they used to have. I find it annoying but practically it's just not a stable state of affairs because it's so easy to change.
I believe the issue is that web development is normally far easier than the alternatives.
So, some ideas are tested in the web, and if they are successful and they benefit from going native, they will be reimplemented in a more close to the metal way. Or not.
GoogleDocs -> Dropbox
VVVVVVV in flash -> VVVVVVVV in C
Some future game in WebGL at 5 FPS -> Same native game at 120FPS with anti-aliasing and anisotropic filtering.
OTOH: Mail clients -> Web mail
So, some ideas are tested in the web, and if they are successful and they benefit from going native, they will be reimplemented in a more close to the metal way. Or not.
GoogleDocs -> Dropbox
VVVVVVV in flash -> VVVVVVVV in C
Some future game in WebGL at 5 FPS -> Same native game at 120FPS with anti-aliasing and anisotropic filtering.
OTOH: Mail clients -> Web mail
This seems to conflict: "The web's great, but this everything-in-the-browser silliness means people are using crappier tools than they used to have."
... with your sentiment on the original post; that it was pointing out how Dropbox integration lets people get out of the browser and use the tools they might be more comfortable with...
Are you for, or against APIs to connect web and desktop and mobile?
... with your sentiment on the original post; that it was pointing out how Dropbox integration lets people get out of the browser and use the tools they might be more comfortable with...
Are you for, or against APIs to connect web and desktop and mobile?
That was unclear. The reason I grouped the dropbox API in there is that it is an HTTP REST API. Good for a lot of things, but to layer a filesystem on top of it is complex and doesn't fit well.
It's a natural extension but it's reinventing the wheel poorly. That happens all the time as things progress. Wheels are reinvented on the new platforms. Normally the new method either pulls ahead quickly or is replaced by an update of the old one. I don't think a good enough implementation of a filesystem over HTTP is easy, and it will at least have to be bloated. A good web aware wrapper around existing filesystem sharing stuff would be much better, sooner of later someone will do that or something even better because they're so clever.
tldr; I like the idea of API's and having internet aware applications for things a browser isn't suited for, and non-http protocols for the things HTTP isn't well suited for. This API strikes me as a poor direction for major development on top of, but makes sense since dropbox is already there and useful.
It's a natural extension but it's reinventing the wheel poorly. That happens all the time as things progress. Wheels are reinvented on the new platforms. Normally the new method either pulls ahead quickly or is replaced by an update of the old one. I don't think a good enough implementation of a filesystem over HTTP is easy, and it will at least have to be bloated. A good web aware wrapper around existing filesystem sharing stuff would be much better, sooner of later someone will do that or something even better because they're so clever.
tldr; I like the idea of API's and having internet aware applications for things a browser isn't suited for, and non-http protocols for the things HTTP isn't well suited for. This API strikes me as a poor direction for major development on top of, but makes sense since dropbox is already there and useful.
All applications in the future will have to be network aware.
They can be web based, but they can use other tech if that fits better.
I believe parent post simply criticizes the fact that some solutions ignore the possibility of something that's not web based.
They can be web based, but they can use other tech if that fits better.
I believe parent post simply criticizes the fact that some solutions ignore the possibility of something that's not web based.
This is not how the Dropbox API works at this moment.
There is no mechanism to push updates to your application. Only the desktop client gets async updates. It will be a glorious day when such an API exists.
But don't hold your breath, it's been a topic on their forums for years.
There is no mechanism to push updates to your application. Only the desktop client gets async updates. It will be a glorious day when such an API exists.
But don't hold your breath, it's been a topic on their forums for years.
Push events to the web app isn't important to me. Syncronization with a web app every few minutes would work well, and you could login to push a button if you needed the changes pulled immediately.
That's the way i implemented it in an iphone app i recently made: a 'sync now' button to pull anything new and push any updates to the server. Works quite well - I recently deleted my app (iphone died) and reinstalled it, one 'sync now' later and all my data was back.
I'm a big fan of using dropbox like this, and am hoping to explore this approach in a lot of productivity apps i plan to make in the future. I like the idea that the user can 'see' their data in a readable format (json,csv,yaml - i still haven't decided which i prefer!) in their dropbox\apps\MyApp folder.
It's like icloud but you feel in control, rather than 'my data is out there somewhere but i can't see it'.
I'm a big fan of using dropbox like this, and am hoping to explore this approach in a lot of productivity apps i plan to make in the future. I like the idea that the user can 'see' their data in a readable format (json,csv,yaml - i still haven't decided which i prefer!) in their dropbox\apps\MyApp folder.
It's like icloud but you feel in control, rather than 'my data is out there somewhere but i can't see it'.
My users would not be this excited if a "sync now" button on the web needed to be pushed. The point of Dropbox integration is to upload photos without leaving Lightroom/Aperture/etc. and not having to install a plugin: http://pgunupudi.blogspot.com/2009/09/dropbox-support-for-ou...
I think this is right on. Dropbox is a virtual file system and not just for the web also for offline. And it works on every platform, I'd much prefer to have things like my contact book and email on Dropbox than where they are today.
I'm guessing the fact that it 'works on every platform' has a lot to due with it's past and future success.
It's a jump back to where we came from in a sense. The desktop file system works well for a lot of business people, which is partly why Dropbox is growing so fast.
It's a jump back to where we came from in a sense. The desktop file system works well for a lot of business people, which is partly why Dropbox is growing so fast.
giving the user that much control, using his example of a spreadsheet on his computer, makes me cringe thinking about how much the user could mess things up with the data.
By contrast, it makes me excited thinking about the possible solutions to the problems you just described. What could be done? Maybe there is a revision system in the web app that shows the changes which must be confirmed before applying? This also would give you web app the ability to easily version control data.
Awesome
Awesome
"How do I undelete files or recover old versions of files?"
http://www.dropbox.com/help/11
http://www.dropbox.com/help/11
Just like with the web application developers will just need to control what a user can do. Users can break native client apps by screwing with config files now but for the most part they don't I think as long as a developer creates some friction (like hidden files) it won't be too much of a problem.
We at http://views.fm completely agree
Looks like your TOS grants you a perpetual all rights license to all content in the user's dropbox without any of the usual qualifications about only as far as it's needed to provide the service?
views.fm: By providing any content to this web site: (a) you agree to grant the site editor a worldwide, royalty-free, perpetual, non-exclusive right and license (including any moral rights or other necessary rights.) to use, display, reproduce, modify, adapt, publish, distribute, perform, promote, archive, translate, and to create derivative works and compilations, in whole or in part. Such license will apply with respect to any form, media, technology already known at the time of provision or developed subsequently;
Dropbox: By using our Services you provide us with information, files, and folders that you submit to Dropbox (together, “your stuff”). You retain full ownership to your stuff. We don’t claim any ownership to any of it. These Terms do not grant us any rights to your stuff or intellectual property except for the limited rights that are needed to run the Services, as explained below.
We may need your permission to do things you ask us to do with your stuff, for example, hosting your files, or sharing them at your direction. This includes product features visible to you, for example, image thumbnails or document previews. It also includes design choices we make to technically administer our Services, for example, how we redundantly backup data to keep it safe. You give us the permissions we need to do those things solely to provide the Services. This permission also extends to trusted third parties we work with to provide the Services, for example Amazon, which provides our storage space (again, only to provide the Services).
I assume that's a cut and paste job gone horribly wrong, but that is pretty gross.
views.fm: By providing any content to this web site: (a) you agree to grant the site editor a worldwide, royalty-free, perpetual, non-exclusive right and license (including any moral rights or other necessary rights.) to use, display, reproduce, modify, adapt, publish, distribute, perform, promote, archive, translate, and to create derivative works and compilations, in whole or in part. Such license will apply with respect to any form, media, technology already known at the time of provision or developed subsequently;
Dropbox: By using our Services you provide us with information, files, and folders that you submit to Dropbox (together, “your stuff”). You retain full ownership to your stuff. We don’t claim any ownership to any of it. These Terms do not grant us any rights to your stuff or intellectual property except for the limited rights that are needed to run the Services, as explained below.
We may need your permission to do things you ask us to do with your stuff, for example, hosting your files, or sharing them at your direction. This includes product features visible to you, for example, image thumbnails or document previews. It also includes design choices we make to technically administer our Services, for example, how we redundantly backup data to keep it safe. You give us the permissions we need to do those things solely to provide the Services. This permission also extends to trusted third parties we work with to provide the Services, for example Amazon, which provides our storage space (again, only to provide the Services).
I assume that's a cut and paste job gone horribly wrong, but that is pretty gross.
I would imagine that a storage company would have something like that if they planned public/social features, such as an imgur for pix you're already storing with them.
Awesome, this feels like a very good idea. But you request full access to sign up? Why not just an app folder?
Cool! What about S3 bucket support?
[deleted]
In terms of these sorts of developer oriented features, I'm much more excited to see Camlistore take off. Portable, better sync options, open source, support for any backend, etc. Just something to look at if this topic is interesting.
In fact, their semi-slogan is "your home directory for the web". Very privacy focused (an encryption layer for backends is in progress based on their issues log), since that seems to get buzz these days.
http://camlistore.org/
In fact, their semi-slogan is "your home directory for the web". Very privacy focused (an encryption layer for backends is in progress based on their issues log), since that seems to get buzz these days.
http://camlistore.org/
Camlistore looks really interesting
[deleted]
Dropbox is merely providing the blueprint for how a cloud a file system should work. I hope they're using their funding to develop something completely above and beyond what they're currently offering, because the network storage space is going to become awfully crowded in a few years.
"...merely providing the blueprint for how a cloud a file system should work."
That sounds like kind of a big deal to me, but hey call me an optimist.
That sounds like kind of a big deal to me, but hey call me an optimist.
As it stands, the only way to receive updates is to poll for changes. This is a kludge at best (and takes away from the magic in a significant way in practice) and completely unrealistic at scale. That is, if I have a million users and I want to update every 15 seconds, Dropbox is going to (rightfully) shut my IP down.
If developers could define WebHooks to receive events, that'd be a great start. I'd happily pay for that.
If developers could subscribe to a WebSockets or event XMPP notification stream of updates, that'd be freaking amazing. I'd stand in line to pay for that. And that's when you'd see an explosion of applications (web, mobile and desktop) implementing Dropbox as their filesystem.
We've been asking nicely for years. If there's anyone here with pull or influence at Dropbox, please give us devs some real-time love. Otherwise, please add your voice to the choir:
http://forums.dropbox.com/topic.php?id=20226&replies=52#...