Own your own data(newsoffice.mit.edu)
newsoffice.mit.edu
Own your own data
http://newsoffice.mit.edu/2014/own-your-own-data-0709
16 comments
Impressive demo. It looks good and everything works ;-)
For those who missed it, there was a thread about Sandstorm two weeks ago: https://news.ycombinator.com/item?id=7961118
For those who missed it, there was a thread about Sandstorm two weeks ago: https://news.ycombinator.com/item?id=7961118
One thing that is missing from most homes is symmetrical internet access. As long as most people can download data significantly faster than upload them, data hosted in the cloud has advantages. Ideally eventually internet connections would become symmetric and fast enough for everyone to have their own low power server with a static ipv6 address at home, with which they can sync all their data. Of course this does not solve the problem that most sever software has complicated configuration and open source alternatives often lack the polish of commercial products. But as soon as there is a switch to ipv6 and symmetric internet access, I see little reason why a decentralized solution would not win.
Why has nobody mentioned https://unhosted.org/ yet?
That's what I thought. They have been around for a while, too, haven't they?
This space is not easy to penetrate. VCs are not interested because there is no easy short term money in it. Large companies are not interested because they would like something with market traction (impressive prototype stage) to buy. Developers who are in this space know well that this requires way more money than you can raise from Kickstarter.
It would be great if all of the people who posted on this page did a google hangout and joined hands (or at least a subset). I think this is bigger than any of these projects individually. Succeeding in this space means something as big as Google, Facebook and Apple. If you are interested in the hangout idea and would like to meet up with other similar project founders (or like minded people), perhaps you can drop me a message (with your email) and we can get started?
The trick is to find a working bootstrapping strategy.
Our approach with sandstorm.io is to make sure it's easy to port existing open source apps. Even with just the platform we have now and the apps we're porting, I feel like Sandstorm is already a product useful enough to stand on its own. Crowdfunding (as we're doing; http://igg.me/at/sandstorm) can't pay for a revolution, but can pay for the MVP, which can then pay for the next iteration, and so on.
"Joining hands" sounds nice in theory but in practice more developers simply does not equal more productivity. At the early experimental stage, it's important to have many small teams iterating quickly on different approaches; a single large team will simply spend all its time arguing and will get nowhere (a lesson I have unfortunately learned the hard way). I am happy to see lots of people working on solutions to this problem because it makes it more likely that one of them will succeed. :)
Our approach with sandstorm.io is to make sure it's easy to port existing open source apps. Even with just the platform we have now and the apps we're porting, I feel like Sandstorm is already a product useful enough to stand on its own. Crowdfunding (as we're doing; http://igg.me/at/sandstorm) can't pay for a revolution, but can pay for the MVP, which can then pay for the next iteration, and so on.
"Joining hands" sounds nice in theory but in practice more developers simply does not equal more productivity. At the early experimental stage, it's important to have many small teams iterating quickly on different approaches; a single large team will simply spend all its time arguing and will get nowhere (a lesson I have unfortunately learned the hard way). I am happy to see lots of people working on solutions to this problem because it makes it more likely that one of them will succeed. :)
Thanks for the comment. I agree that in practice more developers don't equal more productivity. And I agree that things happen better in small teams. But I still don't see the point of not talking to each other, and learning from each other. One reason I suggested what I suggested is this:
If you just run some one else's Linux applications on linux boxes, and call it a personal data store, you have a bunch of problems to solve:
1. No cohesion in UIs 2. No cohesion in APIs 3. A crappy user experience. 4. Code Maintenance
As soon as you think of solving these problems, you'll see the problem of scale, of the number of programmers needed to do things (both frontend and backend).
I still think that the people interested in this should talk to each other(, and perhaps try a divide and conquer approach -- whenever possible), compared to just trying to patch each other's code to make it work in a small team.
I hope you agree, that this is not an easy problem to solve technically. At least I think so.
If you just run some one else's Linux applications on linux boxes, and call it a personal data store, you have a bunch of problems to solve:
1. No cohesion in UIs 2. No cohesion in APIs 3. A crappy user experience. 4. Code Maintenance
As soon as you think of solving these problems, you'll see the problem of scale, of the number of programmers needed to do things (both frontend and backend).
I still think that the people interested in this should talk to each other(, and perhaps try a divide and conquer approach -- whenever possible), compared to just trying to patch each other's code to make it work in a small team.
I hope you agree, that this is not an easy problem to solve technically. At least I think so.
I think all of the problems you bring up are indeed hard problems, and important ones to work on, but they are also problems that exist already, in the current ecosystem of open source web apps, and even often in proprietary apps. About the only way you get "cohesion" on the web today is by limiting yourself entirely to services from one company (Apple; Google), and even then it's far from seamless. Yet, somehow we make progress.
So our goal is not to solve these problems, but to incrementally improve the state of the world. Clearly, the first thing that needs to happen in order for open source and decentralized web apps to be viable is there has to be a way that common, non-technical users can actually use them. Sandstorm is trying to offer a solution for that, and we think we're pretty close. As much as possible, we actually try to stay out of the question of UI or API standards. IMO that's a problem that can only be solved organically.
So our goal is not to solve these problems, but to incrementally improve the state of the world. Clearly, the first thing that needs to happen in order for open source and decentralized web apps to be viable is there has to be a way that common, non-technical users can actually use them. Sandstorm is trying to offer a solution for that, and we think we're pretty close. As much as possible, we actually try to stay out of the question of UI or API standards. IMO that's a problem that can only be solved organically.
Perhaps I don't have enough karma to message directly, but can't see a way to send messages inside HN. Can you provide a way to contact you?
I like the idea of owning and controlling the permissions of all my data at the source, then those settings applying everywhere on the internet - in all the different social media services and systems out there.
Stored in the cloud space of my choice. Continually updated as I live my life, my data would be something I own that I could selectively share to social media services or other individuals or systems with fine-grained control over the whole lot.
Services like Facebook would need to change how they worked. Facebook would not disappear it would just shift its focus to offering a place to use your data.
Taking back the data keys would be a cool direction to evolve, and fun having new responsibilities of managing your own living-breathing data, be it stored in the cloud, your phone, or an encrypted USB stick- up to you.
Stored in the cloud space of my choice. Continually updated as I live my life, my data would be something I own that I could selectively share to social media services or other individuals or systems with fine-grained control over the whole lot.
Services like Facebook would need to change how they worked. Facebook would not disappear it would just shift its focus to offering a place to use your data.
Taking back the data keys would be a cool direction to evolve, and fun having new responsibilities of managing your own living-breathing data, be it stored in the cloud, your phone, or an encrypted USB stick- up to you.
My proposal in this space is unique: decentralized AND built on an existing protocol, email! See:
http://blog.zorinaq.com/?e=76
You don't even need special software to set it up. Standard email autoresponder features make it work (see my demo).
http://blog.zorinaq.com/?e=76
You don't even need special software to set it up. Standard email autoresponder features make it work (see my demo).
Is there some reason this is better than being built on an existing protocol, http!?
"It is quite technical to set up, but in the future if the technology becomes more popular we could imagine webmails making it easier to set up."
The problem is adoption, and this seems to have the same problems as any other approach there, maybe more.
Interesting thought, good luck if you keep pursuing it - sorry to be a downer but it seems like a long-shot IMO.
"It is quite technical to set up, but in the future if the technology becomes more popular we could imagine webmails making it easier to set up."
The problem is adoption, and this seems to have the same problems as any other approach there, maybe more.
Interesting thought, good luck if you keep pursuing it - sorry to be a downer but it seems like a long-shot IMO.
"Is there some reason this is better than being built on an existing protocol, http!?"
Yes: not everybody has a web space to publish HTTP data, but everybody has email. So intuitively building something on email has better chances of gaining adoption.
Yes: not everybody has a web space to publish HTTP data, but everybody has email. So intuitively building something on email has better chances of gaining adoption.
I thought about using smartphones for this kind of thing. They are always online, like a web-server. Also, everyone can install an app.
A server which runs on a smartphone and holds all your data and only friends and services you like, get that data.
The biggest problem would be traffic, I guess, but it could probably be minimized with sending only deltas and p2p-meshs.
This system could use a web-api to integrate better into current landscape.
A server which runs on a smartphone and holds all your data and only friends and services you like, get that data.
The biggest problem would be traffic, I guess, but it could probably be minimized with sending only deltas and p2p-meshs.
This system could use a web-api to integrate better into current landscape.
Having control over your data and paying someone to host the data are not mutually exclusive.
Has someone already coined this movement the "functional cloud"?
This is an idea/concept that has been around for awhile but nobody has been able to execute (well) on it. I like the initial work brought forth through this team and will definitely continue to watch with a close eye.
I think this is a natural progression to the future that the industry doesn't want to happen. Best of luck to the team though and everyone else working on similar projects!
I think this is a natural progression to the future that the industry doesn't want to happen. Best of luck to the team though and everyone else working on similar projects!
This, or something like this, is my best guess at dealing with the "loss" of privacy.
THere are some exciting ideas coming out of (of all places) UK local government, looking at ways to tame the crazy number of proprietary apps that think they should own the database at the centre of their world - simply by forcing the data into the app then back again.
THere are some exciting ideas coming out of (of all places) UK local government, looking at ways to tame the crazy number of proprietary apps that think they should own the database at the centre of their world - simply by forcing the data into the app then back again.
Can you provide a reference to, or a better description of the 'ideas coming out of UK local government' please?
Hard to remember - some devs / architects I met at LGA conference are trying to reduce their internal app footprints and move to a database is not the cow nor yours model.
This is great. Assuming something like this becomes the new normal, I feel like the next problem is micro-payments so that people can get paid for their data.
We have been working on something similar for a few years now: https://register.blib.us
I've spent the last 3 years in this space[1]. 2 of those were with a fellowship from the Shuttleworth Foundation[2].
I applaud all the efforts to free up and decentralize data. I believe it's the future but the way we're approaching it is making that a very distant future.
My suggestions are simple. Data ownership will not be solved by technology. Focus and practice on framing the solution in a way that people connect with. Build that into the fabric of your team. And be prepared to spend a lot of time figuring out how you can describe what a world where you own your data looks like and how it is drastically better than today's world.
It's all too easy to get caught up in writing code.
[1] https://github.com/photo/frontend
[2] https://www.shuttleworthfoundation.org/fellows/current/jaise...
I applaud all the efforts to free up and decentralize data. I believe it's the future but the way we're approaching it is making that a very distant future.
My suggestions are simple. Data ownership will not be solved by technology. Focus and practice on framing the solution in a way that people connect with. Build that into the fabric of your team. And be prepared to spend a lot of time figuring out how you can describe what a world where you own your data looks like and how it is drastically better than today's world.
It's all too easy to get caught up in writing code.
[1] https://github.com/photo/frontend
[2] https://www.shuttleworthfoundation.org/fellows/current/jaise...
> Data ownership will not be solved by technology.
Yes, it might sound disturbing to many, but I'm afraid this is an area which could possibly be push forward massively only by some major government regulation kicking in. In a data-driven economy we live in it would be naive to expect companies to be interested in giving up on user data.
Yes, it might sound disturbing to many, but I'm afraid this is an area which could possibly be push forward massively only by some major government regulation kicking in. In a data-driven economy we live in it would be naive to expect companies to be interested in giving up on user data.
> "Data ownership will not be solved by technology"
I would say 'data ownership will not be solved by technology alone'.
I think it's completely wrong to think we can deal with any of the data ownership issues without understanding how the current technology got us here and what new technologies we need. However, I do agree that throwing some tech out there and expecting it to take off is not going to happen. There is the hard work (as with any new solution) of defining and selling the benefits.
I would say 'data ownership will not be solved by technology alone'.
I think it's completely wrong to think we can deal with any of the data ownership issues without understanding how the current technology got us here and what new technologies we need. However, I do agree that throwing some tech out there and expecting it to take off is not going to happen. There is the hard work (as with any new solution) of defining and selling the benefits.
I agree with that. I think the technology is the easy part. If we knew there was demand for a specific solution then there are plenty of folks who can build it. In fact I think we have built the technology on numerous occasions (i.e. Diaspora, OpenPhoto/Trovebox, OwnCloud, etc.) --- finding the market, describing it in a way that connects with people and being at the right spot at the right time are the difficult parts.
I disagree. The technology needs to be fundamentally different before we can have viable, distributed, alternatives.
We've had crypto for a while and yet nobody encrypts their email by default (or even shares keys). We've had the option to self-host for a long time but few people do. Why? IMHO it's because doing any of these things means you have to become a sysadmin to some degree. Very few people will put up with that for long.
There need to be tools which solve the fundamental (and common) problems of creating distributed systems/applications -- identity, connectivity/sync, deployment. With those tools, new and robust alternatives can be built with the end-user at the centre of their network. Without those new tools, we're simply using band-aids.
We've had crypto for a while and yet nobody encrypts their email by default (or even shares keys). We've had the option to self-host for a long time but few people do. Why? IMHO it's because doing any of these things means you have to become a sysadmin to some degree. Very few people will put up with that for long.
There need to be tools which solve the fundamental (and common) problems of creating distributed systems/applications -- identity, connectivity/sync, deployment. With those tools, new and robust alternatives can be built with the end-user at the centre of their network. Without those new tools, we're simply using band-aids.
> We've had crypto for a while and yet nobody encrypts their email by default (or even shares keys). We've had the option to self-host for a long time but few people do. Why? IMHO it's because doing any of these things means you have to become a sysadmin to some degree. Very few people will put up with that for long.
It's not only that. Previously, encrypting e-mail brought a bigger benefit (at relatively high cost in expertise, maintenance) than it does today, because we no longer own our machines the way we used to. PCs are now permanently connected to the 'net, OS vendors can install anything they like whenever they want, Smartphones come without administrative privileges for the owner even.
What use is encrypted e-mail when various unknown corporate / government entities "own" your device and can work around e-mail encryption easily (by installing a keylogger, grabbing your private key etc.)?
We need to get ownership of our devices back before thinking about crypto too much.
It's not only that. Previously, encrypting e-mail brought a bigger benefit (at relatively high cost in expertise, maintenance) than it does today, because we no longer own our machines the way we used to. PCs are now permanently connected to the 'net, OS vendors can install anything they like whenever they want, Smartphones come without administrative privileges for the owner even.
What use is encrypted e-mail when various unknown corporate / government entities "own" your device and can work around e-mail encryption easily (by installing a keylogger, grabbing your private key etc.)?
We need to get ownership of our devices back before thinking about crypto too much.
> Why? IMHO it's because doing any of these things means you have to become a sysadmin to some degree
I long wished that the App Store model could/would be applied to server stuff.
Imagine the following scenario:
- Buy a Mac Mini (or whatever Airport Server or enhanced Time Capsule)
- get Mail Server.app/Calendar Server.app from the App Store
- have a couple dialogs which configure the sandboxed app with DNS details as provided by DNS provider (which configures a full-blown imap+smtp server, and the remote access) and possibly my email details as provided by my email provider (which configures a fetcher and a smarthost instead)
- physically authorize user devices to accounts via NFC or BT LE/iBeacon. No login/password shingamajig needed! account creation/mapping on the spot!
- download boatloads of personal services from blog to photo management to microblogging to instant messenging to Gitlab to Tor node to whatever innovation came by, some possibly communicating in a decentralized way, possibly without even a need for a DNS record (global zeroconf, DHT, alt DNS, onion routing).
If I can do it with a few debconf-set-selection on dovecot and postfix (plus a few API calls on Gandi to set MX, SPF and whatnot), there's no reason it can't be done automatically for everyone. Of course this is not meant to serve medium to big enterprises (for which the options that actually prevent complete automation exist), but individuals and SOHO really don't need much. People used to think setting up a PC and all its individual apps was a needlessly complicated and/or boring affair (and it was!), now we have built them trivial management. There's no reason our servers could not be treated the same, we just have to stop thinking about the 'old ways' and start with an open mind. I just want you to realise that we tech folks have been doing this for years already just like we did set up and fix computers for everyone for years and we don't have to any more (or way less) thanks to iOS, and Android but also Mac App Store, and soon Chrome Store and Windows Store.
It's a huge endeavor and opportunity to bring such a platform to market, at the right time, with the right pitch, but it has happened before, just on the client-side of things.
I long wished that the App Store model could/would be applied to server stuff.
Imagine the following scenario:
- Buy a Mac Mini (or whatever Airport Server or enhanced Time Capsule)
- get Mail Server.app/Calendar Server.app from the App Store
- have a couple dialogs which configure the sandboxed app with DNS details as provided by DNS provider (which configures a full-blown imap+smtp server, and the remote access) and possibly my email details as provided by my email provider (which configures a fetcher and a smarthost instead)
- physically authorize user devices to accounts via NFC or BT LE/iBeacon. No login/password shingamajig needed! account creation/mapping on the spot!
- download boatloads of personal services from blog to photo management to microblogging to instant messenging to Gitlab to Tor node to whatever innovation came by, some possibly communicating in a decentralized way, possibly without even a need for a DNS record (global zeroconf, DHT, alt DNS, onion routing).
If I can do it with a few debconf-set-selection on dovecot and postfix (plus a few API calls on Gandi to set MX, SPF and whatnot), there's no reason it can't be done automatically for everyone. Of course this is not meant to serve medium to big enterprises (for which the options that actually prevent complete automation exist), but individuals and SOHO really don't need much. People used to think setting up a PC and all its individual apps was a needlessly complicated and/or boring affair (and it was!), now we have built them trivial management. There's no reason our servers could not be treated the same, we just have to stop thinking about the 'old ways' and start with an open mind. I just want you to realise that we tech folks have been doing this for years already just like we did set up and fix computers for everyone for years and we don't have to any more (or way less) thanks to iOS, and Android but also Mac App Store, and soon Chrome Store and Windows Store.
It's a huge endeavor and opportunity to bring such a platform to market, at the right time, with the right pitch, but it has happened before, just on the client-side of things.
Some consumer NAS boxes have this, and there's a Linux distro called Yunohost that does this (and they'd probably love to have more contributors).
From what I understand, a NAS box from Synology actually provides one of the closest experiences to that ideal. It comes with a Package Center[1], which is like an App Store for server applications, and like Android it can install from the standard source or from manually added repositories.
Then you get a panel to configure the new application, usually in a consumer-friendly way, like this iTunes Server[2].
[1] https://www.synology.com/en-us/support/tutorials/500
[2] https://www.synology.com/en-us/dsm/app_packages/iTunesServer...
Then you get a panel to configure the new application, usually in a consumer-friendly way, like this iTunes Server[2].
[1] https://www.synology.com/en-us/support/tutorials/500
[2] https://www.synology.com/en-us/dsm/app_packages/iTunesServer...
I largely feel we should be building companies that do this naturally. If you are making a site where you share photos, bake this concept into your model. Show that it can be non-intrusive for businesses and effectively implemented.
That's what we aimed to do for 2 years before leaving the consumer space altogether. I believe we delivered the user experience and ease of use but we couldn't find the market or the market wasn't big enough.
See my post on why we left the consumer space, https://medium.com/@jmathai/hello-2014-goodbye-consumer-phot...
See my post on why we left the consumer space, https://medium.com/@jmathai/hello-2014-goodbye-consumer-phot...
So, Trovebox has ongoing costs, plus the costs of the storage itself?
We changed the model quite a bit in 2013 when we began focusing on serving the needs of businesses (namely, non-profits).
When we were consumer facing we had a free account with upload limits and an unlimited account for $29/year.
We decided to not differentiate prices based on storage; ours or yours. The pricing page got too complex.
That being said, nearly everything we've done is open source and even when running as a hosted service we decoupled application logic from data storage. That was our goal from the beginning. We went as far as enabling you to switch storage services (i.e move from Dropbox to S3) with a single click.
My mission was to build the technology but in a way non-technical users would understand. I think we accomplished that. Being open source was the underlying prerequisite since it's important if you start on a hosted account you could switch to your own instance or someone else's hosted service.
All that said, I don't think there's a huge market for this in the consumer space today. I hope in time that demand grows because an Internet the way we were envision looks great.
I blogged about leaving the consumer space here, https://medium.com/@jmathai/hello-2014-goodbye-consumer-phot...
When we were consumer facing we had a free account with upload limits and an unlimited account for $29/year.
We decided to not differentiate prices based on storage; ours or yours. The pricing page got too complex.
That being said, nearly everything we've done is open source and even when running as a hosted service we decoupled application logic from data storage. That was our goal from the beginning. We went as far as enabling you to switch storage services (i.e move from Dropbox to S3) with a single click.
My mission was to build the technology but in a way non-technical users would understand. I think we accomplished that. Being open source was the underlying prerequisite since it's important if you start on a hosted account you could switch to your own instance or someone else's hosted service.
All that said, I don't think there's a huge market for this in the consumer space today. I hope in time that demand grows because an Internet the way we were envision looks great.
I blogged about leaving the consumer space here, https://medium.com/@jmathai/hello-2014-goodbye-consumer-phot...
Owning your own data sounds good. Unfortunately, once you share a piece of data, it's out there forever.
What we all need is to have our own personal servers that validate tokens. Then we would just give out these one-time use tokens to people or institutions. Does the bank need a SSN? Well, here is an auto-generated token. Bank stores that, but to validate it, it calls your personal little server, which checks for use.
Unfortunately, systems are built which require personal information. So eventually, for example, the government or a third party service to get credit reports needs your actual SSN. Then you are hosed.
Alternatively, all information could be free, but ALL systems would require your personal server for "permission to use." That of course is highly complicated.
The problem is everybody / every app thinks they need some piece of information. It bothers me when I go to a weekend clinic to take my son and they ask for an SSN. Why? I am going to pay you and walk out of here. "We need to send the report to your real pediatrician." well, just send it. It's not like they wont be able to file it.
I hope somebody smart comes up with a definitive solution, but a lot of processes, people's attitudes and systems need to be recreated from scratch.
What we all need is to have our own personal servers that validate tokens. Then we would just give out these one-time use tokens to people or institutions. Does the bank need a SSN? Well, here is an auto-generated token. Bank stores that, but to validate it, it calls your personal little server, which checks for use.
Unfortunately, systems are built which require personal information. So eventually, for example, the government or a third party service to get credit reports needs your actual SSN. Then you are hosed.
Alternatively, all information could be free, but ALL systems would require your personal server for "permission to use." That of course is highly complicated.
The problem is everybody / every app thinks they need some piece of information. It bothers me when I go to a weekend clinic to take my son and they ask for an SSN. Why? I am going to pay you and walk out of here. "We need to send the report to your real pediatrician." well, just send it. It's not like they wont be able to file it.
I hope somebody smart comes up with a definitive solution, but a lot of processes, people's attitudes and systems need to be recreated from scratch.
The more people complain and refuse to let their SSN be cusomter id, the more vendors will backoff.
OTOH, there are so many "legitimate" SSN abusers that it's a lost cause.
OTOH, there are so many "legitimate" SSN abusers that it's a lost cause.
The US use of SSN seems strange to me, given that the UK equivalent ("NI", national insurance number) isn't nearly as commonly asked for. You need to give it to your employer and sometimes your bank and that's it. Healthcare has its own numbering system, but I've no idea what mine is and it's been years since I needed it for anything.
I suggest viewing a related idea Indie: https://ind.ie
I'm in favour of making it easy for individuals to own their 'digital' life.
However, simply having a 'data store' isn't enough. You need to have a system that actually runs some infrastructure for you and then 'data collection' is an obvious side-effect. There are number of projects I'm involved with that take different approaches to this, including technical infrastructure for distributed systems and personal clouds [1,3] as well as business models and market places [2]. Systems like these can provide benefits to end-users that include resilience and flexibility -- it's not just about privacy.
[1] http://nymote.org/blog/2013/introducing-nymote/ and http://openmirage.org/wiki/overview-of-mirage
[2] http://hubofallthings.com/what-is-the-hat/
[3] https://news.ycombinator.com/item?id=8020635
However, simply having a 'data store' isn't enough. You need to have a system that actually runs some infrastructure for you and then 'data collection' is an obvious side-effect. There are number of projects I'm involved with that take different approaches to this, including technical infrastructure for distributed systems and personal clouds [1,3] as well as business models and market places [2]. Systems like these can provide benefits to end-users that include resilience and flexibility -- it's not just about privacy.
[1] http://nymote.org/blog/2013/introducing-nymote/ and http://openmirage.org/wiki/overview-of-mirage
[2] http://hubofallthings.com/what-is-the-hat/
[3] https://news.ycombinator.com/item?id=8020635
Yep, me too.
Try a demo: https://demo.sandstorm.io/demo
And here's the code: https://github.com/sandstorm-io/sandstorm