Actually, as of 2.4 there is an experimental text search feature built-in: http://docs.mongodb.org/manual/core/text-search/. While it is no Lucene (and is still considered experimental) it does provide simple tokenizing, stop-words and stemming.
One case that could break by this change is if you relied on insert's old default of being a no-op if an object with the same _id already exists. This can be used to efficiently insert a default document without race conditions. Example at https://github.com/RedBeard0531/Mongurl/blob/master/mongurl....
That said, it makes sense for this to be opt-in behavior rather than the default.
Out of curiosity, did you have an index on the ts field that you are matching with?
By the way, if you (or anyone else for that matter) come up with useful benchmarks I'd love to get a copy of them at [email protected]. I have a few of my own, but I'd like to get some real-world workloads from the community to test potential optimizations against.
There is an official tutorial on http://mongodb.org, just click "Try It Out" in the top banner. Karl Seguin also wrote two great tutorials (one on basics, one on geo queries) at http://tutorial.mongly.com/.
Locking in particular has improved substantially since 1.6. For example, 2.0 introduced yielding in some cases where MongoDB would go to disk rather than page-faulting with the lock held[1]. This has been extended and improved for 2.2, along with increasing the granularity of the lock from process-wide to per-db. There are plans to increase the granularity further in future releases.
Take a look at the $push and $addToSet[1] update operators in mongodb. They are both for atomically adding one or more elements to an array in an object. On the other hand, MongoDB doesn't support server-side set union or intersection so if you are using those features redis would still probably be a good use there.
I don't know exactly what your GIS needs are, but it might be worth seeing if MongoDB's geospatial indexing meets them if you are interested in querying free-form data.
True, but since the current version of SM that we use (1.7) isn't thread-safe all calls to it need to be inside of a mutex to prevent trashing global state. It is possible that with a switch to V8 or upgrade to the lastest SM (1.8.5?) it is possible to execute JS without a GIL, however that is something that will need extensive testing to make sure that it works properly.
PS - It is important to note that the jslock is completely separate from the dblock and it is rare to hold both simultaneously. This means that if you are running multiple Map Reduces, one can be fetching or writing data to the DB while the other is processing the objects in JS.
Writing to the journal is actually done outside of the write lock most of the time. It does hold a readlock, however as of 2.0 most commits will release the lock before doing any disk I/O.
2.0 has a new index format that should reduce your index sizes by ~20-30% to fit more in RAM. If you haven't looked at upgrading yet, it is probably worth testing with 2.0.1 to see how it performs in your use case. You will need to reindex() or restore from a dump to take advantage of the new index format.
The global write lock, while important, is probably the most over-written-about and misunderstood issue. The writelock is only held while the db is actually in the middle of an write and tends to only be held for a very brief period (about 10s-100s of microseconds or less) at a time. For example, it is never held across multiple user actions and therefore is closer to a latch than a lock in common RDBMS terminology. The main cases where this caused issues for users are when some data we needed wasn't already in memory so we went to disk to fetch it (~10ms rather than 10us). One of the major improvements to 2.0 was a framework to yield the lock to allow other work while going to disk that is used in the places where this was seen most frequently in production. 2.2 will plug this in more places with the goal to never hold the lock when going to disk. This work is of course in parallel to work to move to DB, collection, and possibly extent-level locking replacing the global lock in almost all cases.
As for safe-mode, the waiting that it does is always outside of the lock. It uses a cached datastructure that is protected by a secondary mutex so that it never has to interact with the global dbMutex. If you choose to wait for replication or journalling of your writes, that will block the connection and therefore your client thread so single-threaded tests will show much worse performance with the db mostly idle. If you use more client threads or asynchronous I/O connections you should see roughly identical throughput in aggregate (see mongostat) although much higher latencies compared with not waiting.
If you have any questions about this feel free to shoot me an email. Replace the _ with an @ and add a .com to my user name.
Even worse is that modern x86 (and probably other) CPUs also have instructions for 16-byte vector registers that can be used to copy or compare data much faster than 1 byte per cycle. Recent versions of glibc use some linker magic to pick the optimal code to use for strcmp, memcpy and friends based on the instruction set available to the CPU at runtime. Of course gcc and glibcxx's developers must not trust glibc and will sometimes replace calls to these functions with thier "optimized" builtin versions that use the lowest common denominator ISA. An easy way we got a 5% boost in througput in mongodb was to force them to call into glibc. https://github.com/mongodb/mongo/blob/master/pch.h#L47-56
Unfortunately a lot of companies don't use today's compilers, but use many-year old versions. For example, Apple still ships gcc 4.2 which is over 4 years old now and supports none of these features.
I think Effective C++ by Scott Meyers[1] is the gold standard when it comes to books on writing good clean C++. It assumes that you already know the basic syntax though and already know how to program. This allow him to skip the intro stuff that take up the most space in many books and get strait to telling you what you need to know to write better code.
Actually since most if not all CPUs have an atomic increment instruction it is quite easy to make shared_ptr<> thread-safe. In fact, that is what boost::shared_ptr<> does. This makes it a good tool for resources that may be shared between threads or for cases where it's just not worth the mental overhead of worrying about ownership.
That said, shared_ptr isn't a solution to everything. Sometimes scoped_ptr (called unique_ptr in tr1) or intrusive_ptr is a better tool. Often if you don't want to share or transfer ownership, passing around a raw ptr or reference is even better and the only option if you want to work with embedded structs (but the newed memory should still be owned by some RAII smart pointer even if it gets passed around as a bare pointer)
That line was referring to restarting just the mongod process on a running machine (eg, when upgrading or changing config flags). In that case, the data should still be in the OS file cache and will result in minor rather than major pagefaults when mongod accesses data so it will not go to disk. The case where you do a full reboot is mentioned two sentences later: "Of course on a full server reboot MongoDB must reheat too."
This gives you much (not all) of the brevity and flexibility of dynamic typing, while maintaining the power and safety of static typing.