I need to add two factoids to this insightful discussion:
There is a big difference between space-filling curves and hyperspace hashing. Space-filling curves map a multidimensional space to a single path through that space that is then mapped to nodes. In the process, they do not retain locality. To our knowledge, hyperspace hashing is a direct intellectual descendant of consistent hashing and has not been done before. If you have pointers to work where data is mapped to nodes in a cluster using a multidimensional hash, please send them to us!
And one major reason why multidimensional databases failed to take off is a problem known as "the curse of dimensionality." If you implement a multi-dimensional representation naively, highly-dimensional data (say, an object with 10-20 attributes) will require a large number of nodes to be efficient. HyperDex solves this through something called space partitioning (I think the paper calls it "data partitioning," but we've changed the name to be a bit more descriptive). They're kind of analogous to materialized views, very loosely speaking.
Agreed completely that hyperspace hashing comes to its own when coupled with value-dependent chaining!
Thanks! Unlike a kd-tree or b-tree variants, HyperDex does not build an auxiliary data structure. It turns out that keeping aux data structures in sync with the data is very difficult if you want to provide strong consistency guarantees. Hyperspace hashing is purely a mapping trick, not a distributed data-structure trick.
Agreed with you fully that value-dependent chains are neat. They allow the system to replicate and relocate data, without any need for background processes. VDCs are the key to HyperDex's strong consistency guarantees.
Not sure when you last looked at it, but we added the ability to cleanly shutdown and restart the cluster two releases ago. We are committed to providing a smooth upgrade path as well; that is not to say we will always be binary-compatible, as the next release involving the disk layer will change the on-disk representation, but we will always provide automatic upgrade scripts. So, if you last tested 0.29b or so, do check out the latest code in the repo.
A website relying on a single service to serve as a sole identity provider seems replete with risks. That sounds remarkably similar to Microsoft's Passport, an idea that was a non-starter for everyone except Microsoft.
Since these kinds of leaks are very difficult to authenticate, there is an opportunity for intelligence agencies to game WikiLeaks by (slightly) altering stolen documents.
I'd take everything in these documents with a huge grain of salt.
You asked for a technical description, and also mentioned that you were unwilling to read a long, detailed technical description, and asked for a "TL;DR." I hope the short summary was useful. If you have technical feedback, we'll be very happy to engage further.
HyperDex is a next generation NoSQL store, so it kind of resembles traditional NoSQL stores like Mongo, Cassandra, Riak, and Dynamo, with a rich interface similar to that of Redis, but it offers much stronger properties than all of these systems. It differs from Mongo in that it provides much stronger consistency and fault-tolerance guarantees. Mongo's default will gladly pretend that an operation was committed even before it was seen by any server. HyperDex differs from Redis in that it shards its data across a network and is generally designed from the ground up for a networked environment. Both Redis and Mongo can and will return stale results following a failure whereas HyperDex will always return consistent results -- it guarantees something called "linearizability" for key-based operations, which roughly means that time will never go backwards from the point of view of any client. And in spite of offering stronger properties, HyperDex is faster than both Mongo and Redis on industry-standard benchmarks.
The HyperDex coordinator sounds like a singular entity, but is in fact a redundant, replicated service. The Paxos algorithm (provided by the ConCoord implementation here: http://openreplica.org) ensures that the coordinator overall can survive failures of some of the coordinator replicas.
Building a pub/sub system on top of HyperDex would be an excellent project.
HyperDex does not support "capped collections" out of the box, but it would be trivial to implement these with a background thread that prunes the database. The data store supports sorted queries, so you can say "return the top-1000 objects sorted by insertion time" or whatever else metric you liked to sort by. And you can delete groups of objects. These operations are implemented efficiently.
A real hacker would never be dismissive of code and claims backed by an open git repo and documentation.
That said, HyperDex outperforms and provides stronger guarantees than previous key-value stores due to two architectural differences.
First is a new way to distribute data called "hyperspace hashing," whose description requires a picture, and can be found here: http://hyperdex.org/about/. This is quite different from the simple bucketing performed by memcached, Dynamo, Cassandra, Riak, MongoDB and others. See the latest slide set for an illustration of differences.
Second, HyperDex maintains replicas through a technique known as "value-dependent chaining." This enables the system to keep the replicas in sync without having to pay high overheads for coordination.
Thanks for the note. We were flabbergasted to read the parent comment complaining about "graphs and numbers."
Note that the issues raised by the Redis developer boil down to the following:
* The benchmark measures a primitive that Redis does not provide, so Redis looks slow: This may be true. The strength of a system lies in how well its primitives handle unanticipated uses. Undoubtedly, a system with a "do_benchmark()" interface would achieve the best results, but this is not a good way to build systems. For the record, HyperDex's interface is at the same level of abstraction as Redis's in this case.
* The benchmark compares "single-core Redis" to "multi-core HyperDex." It is true that HyperDex was designed from the ground up for a networked, multi-core system. Redis is not sharded and seems to work best when co-located on the same host as its clients. If your data fits on one host and clients are on the same machine, you should probably use Redis and not HyperDex. As for the complaint, we would have used something other than "single-core Redis" if such a thing existed. Our emails to Salvatore asking for an alternative binary went unanswered -- he chose to respond with blog entries instead of code.
* The benchmark is not real: The benchmark in question is the Yahoo Cloud Serving Benchmark. It's not something we made up. One can only imagine the kind of criticism we would get if we had actually constructed our own benchmarks. YCSB is an industry-standard benchmark commonly used to evaluate the performance of key-value stores.
These kinds of issues are really easy to resolve, without having to recourse to noise on blogs and HN: We urge everyone to test their own apps against the git repo. We worked hard to make HyperDex the best key-value store out there with the strongest properties, and we hope you find it useful.
There is a big difference between space-filling curves and hyperspace hashing. Space-filling curves map a multidimensional space to a single path through that space that is then mapped to nodes. In the process, they do not retain locality. To our knowledge, hyperspace hashing is a direct intellectual descendant of consistent hashing and has not been done before. If you have pointers to work where data is mapped to nodes in a cluster using a multidimensional hash, please send them to us!
And one major reason why multidimensional databases failed to take off is a problem known as "the curse of dimensionality." If you implement a multi-dimensional representation naively, highly-dimensional data (say, an object with 10-20 attributes) will require a large number of nodes to be efficient. HyperDex solves this through something called space partitioning (I think the paper calls it "data partitioning," but we've changed the name to be a bit more descriptive). They're kind of analogous to materialized views, very loosely speaking.
Agreed completely that hyperspace hashing comes to its own when coupled with value-dependent chaining!