its through the holepunch stack (i am the original creator). Incentives for sharing is through social incentives like in BitTorrent. If i use a model with my friends and family i can help rehost to them
The rust port of Hypercore has been very active recently and they are making good progress. Part of the latest Hypercore release was to move some of the transport crypto to be easier to port to other languages such as rust.
Yea you have to trust the original writers atm. I have some ideas for reducing this trust in the future through some consensus schemes but nothing fully baked yet. Def something I wanna hit tho, so we can get better security in something like a massively distributed chat system.
Hypercore is a single writer append-only log. The website has a bit more info about how it works, but's basically a merkle log signed by a private key / public key scheme. We build collaborative data structures by combining multiple Hypercores.
Hyperdrives builds a p2p filesystem on top of Hypercore for a single writer. Using mounts you can mount other peoples drives so merge conflicts don't happen since there is no overlapping writes.
We are working on a union mount approach as well for overlapping drives (we talk a bit about this in the post)
It's only between the peers in your subset of the swarm for now. They exchange a series of WANT and HAVE messages where they subscribe to the sections of each others logs they are interested in.
We are working on expanding this scheme so peers can help discover peers that have the section you are looking for.
Due to the compressed bitfields these section are quite large. In most cases using a few kilobytes you can share WANT/HAVE for millions of blocks
The peers gossip using compressed bitfields in regards to what data they have. These bitfields are super small so we can pack quite a bit of information.
At the moment we don't do anything special in regards to discovery, but as we scale that's something we want to investigate. Since everything is running on append-only logs we can group the data into sections quite easily so there is some easy wins we can do there with announcing to the dht that you have data in a specific region.
Yes we are continuously exploring and research this. The mount support is our first stepping stone towards this. See the union mounts section of the post
It uses Noise to encrypt all transport connections and using the api you can whitelist which peers (by their public key) you want to replicate with. This API is not yet exposed through the daemon but it’s in the modules.
Hyperdrives also have a built in capability system where you have to know the public key of the drive to download it from a peer, so if you only share the key with yourself no one else can access it.
Finally using the modules you can build almost any kind of networking you’d imagine but that part requires more work on your side obviously.
Yea totally. The FUSE support is already very close to being elexactly that. We’ve been using it internally during our beta as an in office P2P dropbox using the mounts feature
Yep! That is one of the things we’ve worked the hardest on. Completely new indexing structure, using an append-only hash trie which scales really well. We’ve tested it with many big datasets including importing all of Wikipedia as files in a single folder. Worked like a charm :)