Answers to common questions about RethinkDB(rethinkdb.com)
rethinkdb.com
Answers to common questions about RethinkDB
http://www.rethinkdb.com/blog/answers-to-common-questions/
7 comments
slava@rethink here - thanks for setting this up! Is this running on ec2 or on your local infrastructure?
thanks for making it! it's our platform built around heroku's open-sourced toolchain, so we made a little buildpack + app repo w/ Procfile, compiled into a slug and then deployed it out like everything else
Could you shoot me an e-mail to [email protected]? We're looking into building a Heroku plugin, and I'd love to chat about potential solutions to this (we're new to Heroku here).
Thanks, but word of warning - through the Data Explorer interface it is possible to take down a core or two by passing things like 'while (1) {}' to the r.js() command
No good deed goes unpunished, etc.
No good deed goes unpunished, etc.
Hey, it's actually not very difficult for us to make this more robust (e.g. kill and restart the v8 process if it gets out of line). EDIT: here's an issue for this -- https://github.com/rethinkdb/rethinkdb/issues/69
We'll fix this ASAP (though this is somewhat of a low priority).
We'll fix this ASAP (though this is somewhat of a low priority).
There are actually really cool insidious queries you can run. There is for example a query you can run which will insert an infinite number of documents into a table!! Hats off to the first person to post it.
It's awesome that you're hosting this you should just be sure it's on a machine you're not depending on for other services.
It's awesome that you're hosting this you should just be sure it's on a machine you're not depending on for other services.
t.forEach(t.insert(t.without("primaryKey"));
Only try this at home.
Only try this at home.
Thanks. I tried it out and it wasn't too bad. It's in LXC in KVM on hardware, so there's some cgroups and vpus and such in the way of total meltdown
I'm almost fearful for asking this given the somewhat anti-Rails sentiment I sometimes see here, but...
Anyone know of any projects to create a Rails ODM for RethinkDB? (a la Mongoid/MongoMapper?)
Seems to me a natural fit given the (relative) popularity of MongoDB in the Rails world and the comparisons/improvements of RethinkDB over MongoDB.
Anyone know of any projects to create a Rails ODM for RethinkDB? (a la Mongoid/MongoMapper?)
Seems to me a natural fit given the (relative) popularity of MongoDB in the Rails world and the comparisons/improvements of RethinkDB over MongoDB.
Hi Joe Doliner, engineer at RethinkDB here. We've talked a lot about having support for an ODM. It will be happening but it's one of many things we want to add so I'm not certain of a timeline yet. I think the main thing we've discussed is an Active Record plugin. I'm not terribly well versed in the various ODMs that are out there and I'd be interested to know which ones you think are good and why if you wouldn't mind. It will help us in designing one that is awesome.
Well on their about page they have this guy listed:
Michael Lucy (@mlucy) - Ruby drivers, protocol API, query execution
Might be a good place to start.
Might be a good place to start.
To compete with MongoDB for Node crowd, you need an ORM similar to Mongoose. It seems that Node users now default to MongoDB in their stack, much like MySQL in LAMP.
For Golang, mgo http://labix.org/mgo is a quality implementation driver for MongoDB. Hopefully there is something similar for RethinkDB.
For Golang, mgo http://labix.org/mgo is a quality implementation driver for MongoDB. Hopefully there is something similar for RethinkDB.
We're working on improving the API spec so people can write clients easily (we'll publish an RFC soon).
As far as ORMs go, we do plan to do that, but it will take a little bit of time. It shouldn't be very difficult, so we hope to get to it soon.
As far as ORMs go, we do plan to do that, but it will take a little bit of time. It shouldn't be very difficult, so we hope to get to it soon.
Where did you get that from? CouchDB is what I've seen most used and suggested by far.
I've got a question that I haven't been able to find an answer to anywhere.
What would be the best way to set up a one-to-many relationship in RethinkDB? For example, a User has a Store, a Store has Categories, and a Category has Products.
Along the same lines, is there some functionality similar to MongoDB's compound indexes?
What would be the best way to set up a one-to-many relationship in RethinkDB? For example, a User has a Store, a Store has Categories, and a Category has Products.
Along the same lines, is there some functionality similar to MongoDB's compound indexes?
There are two ways to set up a one-to-many relationship. The first is to use an array that holds ids. The second is to use a more traditional db approach and create a separate mapping table where each document holds ids of documents its mapping.
Use the former approach if your array will be relatively small (say, under ~2000 items). Use the latter if the number of relationships is much bigger than that.
As far as compound indexes, we don't have support for that yet, but it will happen.
Use the former approach if your array will be relatively small (say, under ~2000 items). Use the latter if the number of relationships is much bigger than that.
As far as compound indexes, we don't have support for that yet, but it will happen.
I think the latter approach is actually preferable in all cases in which you want to join the 2 (which is normally the whole point). The reason is that it makes it easy to use equijoin which won't work with arrays. So suppose I have table A and table B and table A-to-B describing their relationships. Table A-to-B has documents that look like:
{a_id: "foo", b_id: "bar"}
I can then say: table("A-to-B").eq_join("a_id", table("A")).zip().eq_join("b_id", table("B")).zip()
And you'll get a document for each corresponding pair. If the ids are in an array then it's going to be a bit painful to write that query. (Although it can be done I believe).
A future feature will probably be to make eq_join work with arrays of foreign keys.
I can then say: table("A-to-B").eq_join("a_id", table("A")).zip().eq_join("b_id", table("B")).zip()
And you'll get a document for each corresponding pair. If the ids are in an array then it's going to be a bit painful to write that query. (Although it can be done I believe).
A future feature will probably be to make eq_join work with arrays of foreign keys.
> A future feature will probably be to make eq_join work with arrays of foreign keys.
Nice!!!
Nice!!!
There is also the completely denormalized solution of embedding all children into parents.
This approach could work well when:
- you don't access child documents directly
- you don't update child documents too often
- child documents are usually unique entities
As you can see from the rules above, this would not be a good solution for the Store -> Categories -> Products (contradicts 1. and 3.) But it could work with models like Content -> Votes, Blog -> Comments, etc.
This approach could work well when:
- you don't access child documents directly
- you don't update child documents too often
- child documents are usually unique entities
As you can see from the rules above, this would not be a good solution for the Store -> Categories -> Products (contradicts 1. and 3.) But it could work with models like Content -> Votes, Blog -> Comments, etc.
Good to hear that compound indexes are forthcoming. When you do have secondary index support, will there be functionality to enforce a "unique" index, or will that have to be handled application-side?
Also, in your API documentation [1], I see there's an `append` function to add a value to an array. Is there an equivalent function to remove a value from an array?
[1] http://www.rethinkdb.com/api/#js
Also, in your API documentation [1], I see there's an `append` function to add a value to an array. Is there an equivalent function to remove a value from an array?
[1] http://www.rethinkdb.com/api/#js
Yes, you can do arbitrary slices on an array like in python such as: array[:5]. This is done using the slice command. I believe it's overloaded in python to work with it's native slice syntax.
We can't commit to unique secondary indexes functionality. It depends on a number of technical decisions we are yet to make. We have a rough idea of the architecture, but we still have to work out the specifics, and uniqueness enforcement will depend on that.
What has to happen before rethink is production ready?
I have to build an analytics db for commercial launch in February, it'll be a single machine cluster for the first few months. Would rethink be a risky choice?
I have to build an analytics db for commercial launch in February, it'll be a single machine cluster for the first few months. Would rethink be a risky choice?
slava@rethink here. RethinkDB is new software and still has bugs (see: https://github.com/rethinkdb/rethinkdb/issues) We're working very hard to iron out all reported and known issues, and improve the testing infrastructure to find new ones. There is nothing critical that's wrong with the server that we know of, but I still prefer to wait until the test infrastructure is much more comprehensive before I can comfortably recommend people to put it in production. I suspect this process will take about three months.
That being said, if you choose to build analytics on top of RethinkDB, we'll support you all the way.
That being said, if you choose to build analytics on top of RethinkDB, we'll support you all the way.
I'm building exactly that (it's on postgres now, but I'm considering moving to RethinkDB when it's more stable). It's good to know you have my back.
By the way, for the benefit of the commenters here, I checked RethinkDB two days ago for http://www.instahero.com (the analytics app I'm building), and I ran into a bug where RethinkDB's count() would only return half my data.
The team was very responsive and fixed the bug in a few hours (and pushed the fix so I could test it), so I'm very impressed by the response time. I can't hold the bug against them, because they do say it's not production-ready.
Also, I pointed out that the DB listening to interfaces other than localhost by default was a potential security risk and that, too, was fixed in around a day. Kudos.
By the way, for the benefit of the commenters here, I checked RethinkDB two days ago for http://www.instahero.com (the analytics app I'm building), and I ran into a bug where RethinkDB's count() would only return half my data.
The team was very responsive and fixed the bug in a few hours (and pushed the fix so I could test it), so I'm very impressed by the response time. I can't hold the bug against them, because they do say it's not production-ready.
Also, I pointed out that the DB listening to interfaces other than localhost by default was a potential security risk and that, too, was fixed in around a day. Kudos.
Hi Slava, thanks for that. I'm really excited about rethink and am really looking forward to using it in a serious project.
What make this a ggood fit for analytics backend? How could be better than say, mongo or redis?
"How do I install RethinkDB on other Linux flavors?"
It would be nice to mention that RethinkDB only compiles on x86-64. If you follow the instructions on the linked "building from source" page, it works fine on 32 bit machines ... until you actually run "make".
It wasn't that much time, but I wouldn't have tried if I'd known ahead of time.
Thanks for the effort!
It would be nice to mention that RethinkDB only compiles on x86-64. If you follow the instructions on the linked "building from source" page, it works fine on 32 bit machines ... until you actually run "make".
It wasn't that much time, but I wouldn't have tried if I'd known ahead of time.
Thanks for the effort!
Is it production ready?
No, we're still ironing our some issues, improving the test infrastructure, etc. We'd love for you to use it in personal projects, but I wouldn't put rethink in production quite yet.
Timeline/Roadmap? That is, when will it approach being production ready?
slava@rethink here. I'm hesitant to give a timeline - we won't say the product is production ready until we know it is, and since complex software systems can have bugs lurking under the surface, it's difficult to predict how long it will take.
That being said, the way we're approaching this issue is by overhauling the test infrastructure to stress the server in as many ways as we possibly can. I suspect it will take about three months for us to feel satisfied with how extensive the test infrastructure is. Once that's in place, we'll be able to recommend rethink for production. Of course if the test infrastructure exposes hard-to-kill bugs, it will take a little longer.
That being said, the way we're approaching this issue is by overhauling the test infrastructure to stress the server in as many ways as we possibly can. I suspect it will take about three months for us to feel satisfied with how extensive the test infrastructure is. Once that's in place, we'll be able to recommend rethink for production. Of course if the test infrastructure exposes hard-to-kill bugs, it will take a little longer.
I don't know if this is a common question, but it is a question I have about every data storage and retrieval engine, be it an SQL database or a NoSQL datastore:
How would you re-architect it if you knew ahead of time it would only sit on SSDs - never on spinning media?
How would you re-architect it if you knew ahead of time it would only sit on SSDs - never on spinning media?
Hi Joe Doliner engineer at RethinkDB here. RethinkDB is actually structured in exactly this way because it was originally conceived as a memcached compatible database optimized for solid state drives. Optimizing for solid state drives is a complicated matter on which we published some papers. Here's the HN comment level of detail description:
SSDs are, unlike rotational drives, inherently parallel devices. Each drive has a number of flash memory units which are capable of concurrently doing writes. To split the load evenly we divide files up into zones which we call "extents" the number of extents is configurable and varying the number of concurrent extents will give varying performance for a given drive. Normally there's a sweet spot where you have as many extents as it has independent chips. Extents are written to in a very specific manner. They are append only which means that you start at the top of the extent and fill it up with blocks of a predetermined size. The block size can be tuned to optimize performance as well although I'm not sure off the top of my head which properties of the drive determine which block size is best. Once an extent has been filled up top to bottom we stop writing to it and move on to another extent.
Garbage collection: Each block that we write in an extent is actually a version of a logical block in a btree. When blocks are changes they are rewritten and the old block is marked as garbage. This leads to extents which are filled almost entirely with garbage. When we hit a threshold we look for the extent with the most garbage copy all of the non garbage blocks to an active extent and then reclaim the extent. SSDs have their own garbage collection mechanisms that can seriously degrade the performance of the drive if when they misbehave. If we get the extent size right the drive will have a much easier time of garbage collection (I actually think the drive won't have to do anything at all for gc in many cases.
SSDs are, unlike rotational drives, inherently parallel devices. Each drive has a number of flash memory units which are capable of concurrently doing writes. To split the load evenly we divide files up into zones which we call "extents" the number of extents is configurable and varying the number of concurrent extents will give varying performance for a given drive. Normally there's a sweet spot where you have as many extents as it has independent chips. Extents are written to in a very specific manner. They are append only which means that you start at the top of the extent and fill it up with blocks of a predetermined size. The block size can be tuned to optimize performance as well although I'm not sure off the top of my head which properties of the drive determine which block size is best. Once an extent has been filled up top to bottom we stop writing to it and move on to another extent.
Garbage collection: Each block that we write in an extent is actually a version of a logical block in a btree. When blocks are changes they are rewritten and the old block is marked as garbage. This leads to extents which are filled almost entirely with garbage. When we hit a threshold we look for the extent with the most garbage copy all of the non garbage blocks to an active extent and then reclaim the extent. SSDs have their own garbage collection mechanisms that can seriously degrade the performance of the drive if when they misbehave. If we get the extent size right the drive will have a much easier time of garbage collection (I actually think the drive won't have to do anything at all for gc in many cases.
RethinkDB started as an optimized storage engine for SSDs and the current version is still using the engine. I'm pretty sure someone that worked on the original engine will chime in with more details, but afaik it involved everything from a specialized log-structured disk format and down to kernel hacking. You'll probably find a bit more details in this post old post http://www.rethinkdb.com/blog/on-trim-ncq-and-write-amplific...
We ran into a little bug w/ the ajax requests using the wrong port when run behind an nginx reverse proxy, and the team was very helpful and responsive in getting it fixed: https://github.com/rethinkdb/rethinkdb/issues/63
Seems like a very promising project