Intro to Yada Project - An Open Source p2p Social Network (proof of concept)(yadaproject.com)
yadaproject.com
Intro to Yada Project - An Open Source p2p Social Network (proof of concept)
http://yadaproject.com/
5 comments
Pardon what may seem to be a silly question (amateur "would-be" engineer speaking), but I have a doubt, which is the following:
Data needs to be stored. You either store it locally, and as such it is only accessible when the device is on, or it's stored on a central repository or index, meaning it's always on even if one of the devices is off.
How do you tackle this? How do you make sure that user A tries to access news feed of user B, and fails to get it because user's B device is offline?
Maybe I'm being slow or noobish, but it seems to be the same problem one has with bitorrent: user A downloads from user B if user B is online. If data is common, I have multiple seeds, but social network implies that nobody has the same data. Unless you copy my graph to my friends "device" so one has multiple seeds, which could be quite unpractical on mobile devices if I have a lot of friends.
How do you solve this "intermittent issue" and the synchronization and persistence problems it generates? Without a central "clearing"/indexing server? Even Skype has "Super nodes" to handle this issues...
This aside from the fact that "IPV6 deployment" is not going to be a easy cake to bake, one has to fix the problems faced by GPG trust model to ensure that two parties can verify (with some degree of trust) that the other party with whom they're communicating is, indeed, the person they expect, dynamic IPs, and so on.
Data needs to be stored. You either store it locally, and as such it is only accessible when the device is on, or it's stored on a central repository or index, meaning it's always on even if one of the devices is off.
How do you tackle this? How do you make sure that user A tries to access news feed of user B, and fails to get it because user's B device is offline?
Maybe I'm being slow or noobish, but it seems to be the same problem one has with bitorrent: user A downloads from user B if user B is online. If data is common, I have multiple seeds, but social network implies that nobody has the same data. Unless you copy my graph to my friends "device" so one has multiple seeds, which could be quite unpractical on mobile devices if I have a lot of friends.
How do you solve this "intermittent issue" and the synchronization and persistence problems it generates? Without a central "clearing"/indexing server? Even Skype has "Super nodes" to handle this issues...
This aside from the fact that "IPV6 deployment" is not going to be a easy cake to bake, one has to fix the problems faced by GPG trust model to ensure that two parties can verify (with some degree of trust) that the other party with whom they're communicating is, indeed, the person they expect, dynamic IPs, and so on.
You can have your data hosted if you don't believe your device will be online all the time. This could open up some ad-based services.
I don't think that answers my question on intermittence, persistence and syncronization.
I can have my date hosted, but if my friends don't (that's why they use your service, so not to have to do that, or else they remain with Facebook, LinkedIn, etc. That's your USP), I can only access their data when my friends devices are online. Or, alternatively, you store my whole friends graph on my device, which presents scalability and performance issues. Or, 3rd route, you store my friends data on multiple devices on the network, doing "multiple seeds". Again, we're back to "opps my data is all over the place", when your Unique Selling Point was P2P = privacy, no need to centralize.
Even Skype Supernodes don't solve this issue. They solve some of the routing issue. Not the "address book/data issue". You don't seem to solve it either.
I can have my date hosted, but if my friends don't (that's why they use your service, so not to have to do that, or else they remain with Facebook, LinkedIn, etc. That's your USP), I can only access their data when my friends devices are online. Or, alternatively, you store my whole friends graph on my device, which presents scalability and performance issues. Or, 3rd route, you store my friends data on multiple devices on the network, doing "multiple seeds". Again, we're back to "opps my data is all over the place", when your Unique Selling Point was P2P = privacy, no need to centralize.
Even Skype Supernodes don't solve this issue. They solve some of the routing issue. Not the "address book/data issue". You don't seem to solve it either.
"Or, alternatively, you store my whole friends graph on my device, which presents scalability and performance issues."
With all due respect, this is where you are incorrect. Devices today completely capable of handling a large amount of compressed plain text data. Media would will be links to external sources as storage capacities play catch up.
The CPU of the device would be able to map your entire social graph and do things that are not possible today being that calculating a social graph on a central server for each user is EXTREMELY expensive and basically impossible with high-load services. Kind of like the concept behind folding @ home.
With all due respect, this is where you are incorrect. Devices today completely capable of handling a large amount of compressed plain text data. Media would will be links to external sources as storage capacities play catch up.
The CPU of the device would be able to map your entire social graph and do things that are not possible today being that calculating a social graph on a central server for each user is EXTREMELY expensive and basically impossible with high-load services. Kind of like the concept behind folding @ home.
There appear to be a number of problems with this proof of concept:
1. A requirement seems to be the ability to make and receive socket connections. How would you address the typically restrictive home firewalls/NATs that most people are living with?
2. In the responses, a commenter raises this issue: With the above example of two users and 56 connections it seems like there could be scaling issues and or perormance issues. They may be less obvious on computers but mobile devices with more limited bandwidth and sometimes quotas may suffer for those users with muilple devices and their multi device highly connected friends.
The response from admin is: "Correct, this technology will have to scale along with hardware. I don’t believe we are too far off with 4g, more inexpensive memory and the like."
That doesn't inspire much confidence in the idea as a proof of concept.
You're also claiming to solve a far more fundamental problem: that of creating and maintaining a reliable distributed peer-to-peer network without any central servers at all. If you do have a practical solution to this problem, then a bunch of other difficult problems (like tracker-less Bittorent, for example, which currently relies on distributed hash tables) reduce to yours. Are you sure you're able to make that claim?
1. A requirement seems to be the ability to make and receive socket connections. How would you address the typically restrictive home firewalls/NATs that most people are living with?
2. In the responses, a commenter raises this issue: With the above example of two users and 56 connections it seems like there could be scaling issues and or perormance issues. They may be less obvious on computers but mobile devices with more limited bandwidth and sometimes quotas may suffer for those users with muilple devices and their multi device highly connected friends.
The response from admin is: "Correct, this technology will have to scale along with hardware. I don’t believe we are too far off with 4g, more inexpensive memory and the like."
That doesn't inspire much confidence in the idea as a proof of concept.
You're also claiming to solve a far more fundamental problem: that of creating and maintaining a reliable distributed peer-to-peer network without any central servers at all. If you do have a practical solution to this problem, then a bunch of other difficult problems (like tracker-less Bittorent, for example, which currently relies on distributed hash tables) reduce to yours. Are you sure you're able to make that claim?
"1. A requirement seems to be the ability to make and receive socket connections. How would you address the typically restrictive home firewalls/NATs that most people are living with?"
IPv6 is already being deployed from top tier mobile carriers and that allows globally rout-able addresses even behind NATs.
"That doesn't inspire much confidence in the idea as a proof of concept."
This technology is a little ahead of its time, but not by much. And by the time an end-user ready core is developed, we'll we be there.
"Are you sure you're able to make that claim?"
yes, the technical hurdles are only time consuming, not insurmountable.
IPv6 is already being deployed from top tier mobile carriers and that allows globally rout-able addresses even behind NATs.
"That doesn't inspire much confidence in the idea as a proof of concept."
This technology is a little ahead of its time, but not by much. And by the time an end-user ready core is developed, we'll we be there.
"Are you sure you're able to make that claim?"
yes, the technical hurdles are only time consuming, not insurmountable.
> IPv6 is already being deployed from top tier mobile carriers and that allows globally rout-able addresses even behind NATs.
With all due respect, if your solution depends on the following developments in the real-world:
(1) Everyone upgrades their cheap DSL/cable home modems to something that supports IPv6 (2) Cell carriers allow incoming mobile data connections (can you say mobile DoS?) (3) People modify their firewalls (again, nothing to do with IPv4 or IPv6) to allow incoming connections (4) You devise some way of allowing static or otherwise identifiable IPv6 IPs, even when the ISP doesn't give you a static IPv6 IP
then I would say you have a number of challenges ahead of you. Assuming that device X and connect to device Y on the Internet, for consumer devices, is a very difficult proposition.
With all due respect, if your solution depends on the following developments in the real-world:
(1) Everyone upgrades their cheap DSL/cable home modems to something that supports IPv6 (2) Cell carriers allow incoming mobile data connections (can you say mobile DoS?) (3) People modify their firewalls (again, nothing to do with IPv4 or IPv6) to allow incoming connections (4) You devise some way of allowing static or otherwise identifiable IPv6 IPs, even when the ISP doesn't give you a static IPv6 IP
then I would say you have a number of challenges ahead of you. Assuming that device X and connect to device Y on the Internet, for consumer devices, is a very difficult proposition.
Thanks for the reply, but take Skype for example, they have gotten over these technical hurdles. I could give you a huge explanation involving relay points and NAT traversal etc. but at the end of the day, these challenges have been defeated.
Skype uses a number of super-nodes that provide some guarantees on service level and has a set of central login servers:
http://www.mjalali.com/blog/?p=10
Yes, through mutual friends and indexers with persistent connections, these challenges are mitigated.
> I could give you a huge explanation involving relay points and NAT traversal etc. but at the end of the day, these challenges have been defeated.
[citation needed]
All of them (to my knowledge, and yes, I've researched this) still require some sort of central network that gets contacted first. "Trackerless" bit torrent, for example, uses a DHT that is hosted by... bittorrent.com
[citation needed]
All of them (to my knowledge, and yes, I've researched this) still require some sort of central network that gets contacted first. "Trackerless" bit torrent, for example, uses a DHT that is hosted by... bittorrent.com
Indexers assist with making connections but the connections are still eventually happening directly.
If you intend to implement this, I suggest you do some serious in depth reading on peer discovery over IP based networks, including real world applications and exactly how they operate, and what their limitations are.
With all due respect, based on your posts your current understanding of how things like Skype and Bittorrent operate under the hood in a "server-less" environment has a few holes.
http://en.wikipedia.org/wiki/Distributed_hash_table
http://en.wikipedia.org/wiki/Local_Peer_Discovery
http://en.wikipedia.org/wiki/Decentralized_network_42
http://samy.pl/pwnat/
With all due respect, based on your posts your current understanding of how things like Skype and Bittorrent operate under the hood in a "server-less" environment has a few holes.
http://en.wikipedia.org/wiki/Distributed_hash_table
http://en.wikipedia.org/wiki/Local_Peer_Discovery
http://en.wikipedia.org/wiki/Decentralized_network_42
http://samy.pl/pwnat/
Have you spoken with the Diaspora team? This was their exact idea, until they started implementing, and the practical concerns moved them back to the client-server, node-based model.
There model is very different, though I don't know their history.
Their model is very different now, yes, but their original presentation was based on this exact concept. I would suggest looking at their history to see why they moved away from this design, and if possible, speaking with them or discussing on their mailing list the pitfalls they ran into.
I'll speak with anyone willing to help solve technical challenges associated with this model. But until someone comes up with something "damning" then I'm going to keep moving forward.
I wish you the best of luck, and I admire your ambition (and hubris) but as someone who has been working on distributed social networking (Appleseed) for over half a decade, I'll present this:
You don't always need a singular, damning situation in order to prevent a model from being implemented. Sometimes dozens of purgatorial ones are plenty.
You don't always need a singular, damning situation in order to prevent a model from being implemented. Sometimes dozens of purgatorial ones are plenty.
It just takes time and dedication, the motto of any true software developer.
Using the CPU power of a device that is capable of playing video games to calculate your social graph (that is stored on that same device) is extremely powerful and well worth the time and dedication.
Using the CPU power of a device that is capable of playing video games to calculate your social graph (that is stored on that same device) is extremely powerful and well worth the time and dedication.
Sorry, but I don't understand how this works without a central server.
edit: Namely, how do you securely add a friend without prior knowledge of his public key?
edit: Namely, how do you securely add a friend without prior knowledge of his public key?
Checkout the how-to page and watch that video. It gives more detail.
But in short, you can make friends with your friend's friends or you can be friends with an indexer, which will list a bunch of people you may or may not know that you can request to be friends with as well.
When you send a request through a friend, your connection is done securely through that already established relationship. Then that friend passes the request to the friend you want to be friends with that they are already friends with. Once that request reaches that friend and they accept, they will have the public / private key pair you originally generated. They will also have your IP address information and begin making secure connections directly to you without the need for the friend or indexer which you sent the request through.
But in short, you can make friends with your friend's friends or you can be friends with an indexer, which will list a bunch of people you may or may not know that you can request to be friends with as well.
When you send a request through a friend, your connection is done securely through that already established relationship. Then that friend passes the request to the friend you want to be friends with that they are already friends with. Once that request reaches that friend and they accept, they will have the public / private key pair you originally generated. They will also have your IP address information and begin making secure connections directly to you without the need for the friend or indexer which you sent the request through.
How do you intend to tackle NAT, dynamic IP addresses, and devices that aren't always on, or devices only accessible to some machines but not others?
How do you intend to solve the chicken and the egg problem this network will encounter?
How do you intend to solve the chicken and the egg problem this network will encounter?
IPv6 is already being deployed from top tier mobile carriers and that allows globally rout-able addresses even behind NATs.
If you have no social graph on yada, then your very first connection to a friend may have to be unencrypted, but not in every case.
If you have no social graph on yada, then your very first connection to a friend may have to be unencrypted, but not in every case.
> IPv6 is already being deployed from top tier mobile carriers and that allows globally rout-able addresses even behind NATs.
You're making a lot of assumptions:
1) That IPv6 will assign a unique and constant IP to every device. Many ISPs provide this as an up-sell feature, not the default, and I don't expect them to change their behavior even if the cost of it approaches 0.
2) That IPv6 will solve all your network problems. It doesn't solve the issue of a bad link or a restrictive firewall. Consider Hosts [A,B,C] and Firewall F. F will block all connections from B. How will yada connect A and B in this arrangement: A <-F-> [B,C] ? Will is use C to route the connection? How will it determine that C can act as an intermediary, and how will it protect A's data such that C can not be used as a MTM (Man in The Middle) attack vector?
You're making a lot of assumptions:
1) That IPv6 will assign a unique and constant IP to every device. Many ISPs provide this as an up-sell feature, not the default, and I don't expect them to change their behavior even if the cost of it approaches 0.
2) That IPv6 will solve all your network problems. It doesn't solve the issue of a bad link or a restrictive firewall. Consider Hosts [A,B,C] and Firewall F. F will block all connections from B. How will yada connect A and B in this arrangement: A <-F-> [B,C] ? Will is use C to route the connection? How will it determine that C can act as an intermediary, and how will it protect A's data such that C can not be used as a MTM (Man in The Middle) attack vector?
1) Each client checks for internal and external IPs and saves them. That list is transmitted to your friends every time you update them with other data. It automatically manages this list of possible places to reach you.
2) There will be indexers for these cases which can be used as relay points. C would merely be a match-maker, telling each host which IP address and port is currently open. No data is sent through C.
Skype does this.
2) There will be indexers for these cases which can be used as relay points. C would merely be a match-maker, telling each host which IP address and port is currently open. No data is sent through C.
Skype does this.
Skype doesn't do this. Please read: http://www.mjalali.com/blog/?p=10
>1) Each client checks for internal and external IPs and saves them. That list is transmitted to your friends every time you update them with other data. It automatically manages this list of possible places to reach you.
According to the link, "supernodes" route connections and provide matchmaking between Skype callers. How do you find the "supernodes"? "If this file is not present, SC tries to establish a TCP connection with each of the seven Skype maintained default SNs IP address on port 33033."
>2) There will be indexers for these cases which can be used as relay points. C would merely be a match-maker, telling each host which IP address and port is currently open. No data is sent through C.
"If caller and receiver are behind a UDP-restricted firewall they will need a relay (node) in between to establish TCP connection to and then the traffic (including media) will go through from one side to the other."
Skype's protocol would use host C as an intermediary.
>1) Each client checks for internal and external IPs and saves them. That list is transmitted to your friends every time you update them with other data. It automatically manages this list of possible places to reach you.
According to the link, "supernodes" route connections and provide matchmaking between Skype callers. How do you find the "supernodes"? "If this file is not present, SC tries to establish a TCP connection with each of the seven Skype maintained default SNs IP address on port 33033."
>2) There will be indexers for these cases which can be used as relay points. C would merely be a match-maker, telling each host which IP address and port is currently open. No data is sent through C.
"If caller and receiver are behind a UDP-restricted firewall they will need a relay (node) in between to establish TCP connection to and then the traffic (including media) will go through from one side to the other."
Skype's protocol would use host C as an intermediary.
That's true, its not exactly the same as your example. But the method I have developed does work.
You didn't address my point about initial discovery being maintained by Skype.
Also, could you please point out to me the method you have developed for initial discovery? I've yet to see it in your source code. This is a very tough nut to crack and if you've got it done already then I'm impressed.
Also, could you please point out to me the method you have developed for initial discovery? I've yet to see it in your source code. This is a very tough nut to crack and if you've got it done already then I'm impressed.
And if you're impressed, what happens? Do you help me? Because I'll fight tooth and nail to impress you if you'll help me.
Seriously.
Seriously.
Show me one place in your code right now where you have done something innovative with peer discovery, that no network engineer, or researcher has tried before, doesn't use a central meeting point, and is successful at discovering peers in a reasonable amount of time. I'll start work immediately. I'm pretty sure more people will join in on it too if this is the case.
That is not coded at this time but you shall see, sir.
Is that a joke? Seriously?