Poor performance from your MongoDB database? Look at the shard key(gobitcan.com)

21 pointsby aggssvor 12 Jahren15 comments

Poor performance from your MongoDB database? Look at the shard key

http://gobitcan.com/blog/2014-12-05-poor-performance-from-your-mongodb-database-look-at-the-shard-key

4 comments

23davidvor 12 Jahren

For startups relying on Mongodb for critical infrastructure, the cost of having a poorly implemented mongodb cluster can actually be high enough from a morale and product performance standpoint to ruin the company. I've seen it happen a few times to very well funded companies where their inability to quickly solve MongoDB issues made it impossible to continue iterating on their product, and eventually caused most of their core backend team members to quit.

So I think it's super important to de-risk MongoDB as quickly as possible and then move on with life. If you can use bare metal, just switch to using SSDs or even FusionIO cards. The new AWS SSD instance types should also be ok. Pay the money you need to service providers like AWS, Rackspace or Softlayer etc. to get it done.

In a best-case situation, you can keep upgrading every 12-18 months and rely on Moore's law to allow your MongoDB machines to scale with your growth. But even in a bad situation where your database is growing in size too fast, this vertical scaling strategy should still give you 6-12 months of breathing room, to allow your dev team to cleanly switch over to a different database.

I see a lot of teams starting out with MongoDB, but the smart ones know it's a big risk and once they find market fit, their first major initiative is to switch over the database backend to use something like Cassandra + Mysql.

pbbakkumvor 12 Jahren

Not unique to Mongo, other systems that shard based on a user-generated key (HBase) have issues with this too. We developed this UUID generator as an attempt at a more thoughtful shard key: https://github.com/groupon/locality-uuid.java

beck5vor 12 Jahren

Unless I am missing something but dumping your entire db then reimporting would take hours in most cases. Staying available during that time would be a nightmare.

rspeervor 12 Jahren

This fits my experience with distributed MongoDB -- which I never tried to use beyond 1.8 or so. The moment you make a MongoDB distributed, you're putting an immutable schema on it.

Not literally immutable, because you can change it, but changing it is more difficult and more perilous than starting from scratch.

Wasn't the point of MongoDB was supposed to be that it's all flexible and schemaless? You might ask that. You might not get an answer. Like I said, I never use MongoDB as a distributed system anymore.

mrkurtvor 12 Jahren

You're not missing something. The dump / restore is the operationally simple answer, but not the best for availability.

phpnodevor 12 Jahren

This is not meant as a troll post but a genuine observation after spending the last year or so migrating various businesses away from MongoDB (or from parse which uses mongo):

Has anyone ever built a non trivial application on MongoDB and been happy with their decision, say, 1 year later? If so, what kind of use case was it? I am genuinely curious.

The impression I've received is that Mongo is ridiculously easy to get started with but the flaws become more and more of a hindrance over time, eventually making certain types of application features virtually impossible to implement.

ecaronvor 12 Jahren

Absolutely. We've been using Mongo for all the core functionality at TrackIf.com and are still very happy with it. (Our biggest complaint was our initial MongoDB host provider was hosing our performance, once we switched to a non-shared Mongo environment it was a night-and-day difference.)

phpnodevor 12 Jahren

is it fair to guess that you use elasticsearch for product search and mongo for tracking item level prices? i.e. most of your mongo queries are direct key lookups?

ecaronvor 12 Jahren

Probably 10% of our queries are direct key lookups, the rest easy-to-read yet still non-trivial.

collywvor 12 Jahren

I would love to see a poll of this (with developers experience included). Two or three years back everyone was ranting about how great Mongo was. Now most articles I see seem to be about how bad it is. And I see lots of developers squeezing relational problems into Mongo. I assume it is good for some things, but it would be interesting to see where people have had problems and why.

bsg75vor 12 Jahren

Its apparent popularity and simultaneous vilification is an interesting contrast.

akbar501vor 12 Jahren

Doesn't eBay use MongoDB at scale (ref: http://www.slideshare.net/jaykumarpatel/cassandra-data-model...).

Slide 4 shows eBay's MongoDB cluster size: Hundreds of nodes, > 50TB, > 2 billion ops/day

phpnodevor 12 Jahren

thanks for the link, I wonder how they use it? The companies I've been working with have made the mistake of using Mongo as their sole data store for everything, rather than for one specific purpose.

akbar501vor 12 Jahren

Media metadata is one use case:

http://www.mongodb.com/presentations/storing-ebays-media-met...

Here's another deck from 2012: http://www.slideshare.net/mongodb/mongodb-at-ebay

phpnodevor 12 Jahren

sounds like more direct key lookups. I can definitely believe mongo is suited for that particular use case, but I'm super sceptical about it being used as a general data store.