Spanner stores its data in Colossus, so there would be some bootstrapping issues to resolve to move it to Spanner over Bigtable. (Bigtable also has the bootstrap issues but has solved them already and there are additional difficulties due to some details that I probably am not at liberty to share)
Spanner is used for metadata for many other very very large storage systems, though.
It's true that the project was initially developed at Cloudera, and employees continue to be the main driving force behind development. That said, we have committers and contributors from other companies as well. Roughly half the people who contributed a patch in the last 3 months have been non-Cloudera. Additionally we are very strict about doing all development upstream (eg with the first open source release we spent a lot of effort to open the entire development history going back to 2012, including JIRA, git, etc).
As for users, here are a couple examples off the top of my head who aren't currently paying for any support:
Hopefully this isn't too "pitch"-y, but: if you're looking for a database that's good at time series, will always be open source, and does support scale-out and HA, you might be interested in Apache Kudu (incubating).
FYI on the date issue, lest anyone think we filed the patent trying to steal the work done by others, the patent application says:
"This application claims to the benefit of U.S. Provisional Patent Application No. 61/911,720, entitled “HYBRIDTIME and HYBRIDCLOCKS FOR CLOCK UNCERTAINTY REDUCTION IN A DISTRIBUTED COMPUTING ENVIRONMENT”, which was filed on Dec. 4, 2013, which is incorporated by reference herein in its entirety."
(which predates the creation of the cockroachdb repo and the hybrid logical clock paper).
Agreed -- personally I'm against offensive use of patents like this as well, and it's my understanding that Cloudera doesn't intend to use this patent offensively. If it did, I would be upset and would consider leaving the company - I know many other employees feel the same way. The reason I agree to help write patent applications as an engineer is that I've seen the distraction and damages caused by patent trolls (or even other companies) and the importance of having a defensive portfolio.
Disclaimer: Obviously I'm not speaking for the company or making any promises here :)
Typically QSBR algorithms don't require blocking the world, or even blocking any single thread. They just require each thread to periodically check in and run a bounded amount of code which amounts to "hey, I'm not currently looking at the map".
Some other background collector thread (which is going to actually delete removed objects) just has to wait until it sees every mutator thread cross a safepoint, at which point it knows that none of those threads could be hanging onto references that have been unlinked from the data structure.
I'd recommend reading some surveys of RCU and SMR algorithms if this stuff is interesting to you.
We've just updated to C++11 as of a couple weeks ago. Partially the issue was when it started, partially the issue is that we have to support older platforms and we have a C++ client library to worry about.
Basically the short answer, though, is that reads are not done by quorum, but the client can specify if they need "up to date" reads. If they do, currently, we force reads to go to the leader, but have a roadmap for how to read from other replicas (see the Spanner paper for a rough idea how that can work).
It doesn't yet. It's on our nebulous "we'd like to do this some time" roadmap, but currently concentrating on some more basic stuff around stability and time series features.
Of course this is a huge optimization for data warehousing applications, where two co-partitioned tables can be joined without any network data transfer, and in some cases could even use merge join instead of hash based strategies. But, it's the usual time/scope/quality trinity, and we'd rather not compromise the third element.
Todd from the Apache Kudu (incubating) team here. I'll check this thread throughout the day in case there are any questions (and try to check the original post for comments as well).
Yep, I've been taking part in those design discussions. We hope to have Kudu tablet servers support generating this in-memory format in shared memory as the result of scans, so the Impala server (client from Kudu's perspective) can directly operate on the data. We're expecting a 20-30% speed boost from this for some queries, though haven't done any tests at scale of the prototype.
Yep, that's correct. HDFS+Parquet is more accurate but doesn't fit quite as well on slides and short descriptions.
The idea is to get the analytic scan performance of Parquet while still allowing for in-place updates and row-by-row access like HBase.
HDFS (with Parquet or other formats) will still be better for unstructured or fully immutable datasets. HBase will still be better when your top priority is ingest rate, random access, and semi-structured data. Kudu should be good when you've got tabular data as described above.
Spanner stores its data in Colossus, so there would be some bootstrapping issues to resolve to move it to Spanner over Bigtable. (Bigtable also has the bootstrap issues but has solved them already and there are additional difficulties due to some details that I probably am not at liberty to share)
Spanner is used for metadata for many other very very large storage systems, though.