Mongo Vs Redis, The Increment Battle(devdazed.com)
devdazed.com
Mongo Vs Redis, The Increment Battle
http://devdazed.com/post/2737860983/mongo-vs-redis-the-increment-battle
12 comments
I wouldn't say it's completely useless as it shows that Mongo is faster in a single threaded process, using the most popular ruby client of each system.
If I were writing a high-concurrency multi-threaded system to handle millions of keys at a time, Ruby would not be my first choice by far.
If I were writing a high-concurrency multi-threaded system to handle millions of keys at a time, Ruby would not be my first choice by far.
> it shows that Mongo is faster in a single threaded process, using the most popular ruby client of each system.
No, it doesn't. You ran 100k increments on one key and divided by time ONCE and printed the result.
Plus this was on your Macbook Pro, which is probably running a mail client, Twitter client, web browser, and hundreds of other things that could have affected the times.
Graphing the response times for each increment would be a good way to start looking at what the data really shows, instead of going by a single opaque, fairly useless number.
No, it doesn't. You ran 100k increments on one key and divided by time ONCE and printed the result.
Plus this was on your Macbook Pro, which is probably running a mail client, Twitter client, web browser, and hundreds of other things that could have affected the times.
Graphing the response times for each increment would be a good way to start looking at what the data really shows, instead of going by a single opaque, fairly useless number.
I have to agree with antirez here and call shenanigans. "WOW, MongoDB is almost a full 40% faster at incrementing than Redis" is simply a false statement. The correct conclusion for your post is "WOW, incrementing the same key 100k times with the mongo ruby gem in a very specific, single-threaded ruby application is 40% faster than incrementing one key 100k times with the redis ruby gem the in that same ruby application." If anyone makes a decision about a real world application based on your post, they will be making a huge mistake.
Check this comment: http://news.ycombinator.com/item?id=2103945
it could even be a dummy server just replying +OK and the result will not change.
You should try this against MySQL too, and you'll see similar numbers, for sure not an order of magnitude different than that even if SQL involves much more query processing and so forth.
it could even be a dummy server just replying +OK and the result will not change.
You should try this against MySQL too, and you'll see similar numbers, for sure not an order of magnitude different than that even if SQL involves much more query processing and so forth.
Ok, I agree with antriz's statement. My results are very specific. I will make note and be sure to be very thorough next time around.
Thanks for agreeing, however I must point out that since your blog post does not accept comments, nor you rectified the content, this is misleading information that will stay on the internet for virtually many years to come.
More accurate title:
(Mongo + Mongo Ruby Gem) Vs (Redis + Redis Ruby Gem), The Increment Battle with No Useful Statistical Analysis Included
(Mongo + Mongo Ruby Gem) Vs (Redis + Redis Ruby Gem), The Increment Battle with No Useful Statistical Analysis Included
You could make electricity from the amount of eyerolling this article is going to generate here on HN.
I think I heard a distant scream of Zed... http://zedshaw.com/essays/programmer_stats.html He was so right for cases like this...
I'm a programmer, but not very good at statistics (as in I passed the class with flying colors, but I can't really comment on new situations).
When it comes to things like benchmarking, I find it easiest to avoid making absolute conclusions like "X is faster than Y", but to merely present the data, all the data, and nothing but the data.
In this case, I'd have wanted to see the response times per request, and perhaps the memory trend for the client. At the very least, I'd like to know how Mongo and Redis were setup, and if either had begun to swap.
When it comes to things like benchmarking, I find it easiest to avoid making absolute conclusions like "X is faster than Y", but to merely present the data, all the data, and nothing but the data.
In this case, I'd have wanted to see the response times per request, and perhaps the memory trend for the client. At the very least, I'd like to know how Mongo and Redis were setup, and if either had begun to swap.
Running your benchmark I get 11216 increments per second
Running the redis-benchmark from (http://redis.googlecode.com/files/redis-2.0.4.tar.gz) with 50 parallel clients 37800 requests per second
I would like to see the numbers for 50 clients to mongo doing the same.
Running the redis-benchmark from (http://redis.googlecode.com/files/redis-2.0.4.tar.gz) with 50 parallel clients 37800 requests per second
I would like to see the numbers for 50 clients to mongo doing the same.
interesting but 37800 is low too, either not Linux or is a Virtual Machine :) That's what I get in my desktop system with Linux (Xeon E5520 2.27 Ghz):
INCR: 126743.98 requests per second
INCR: 126743.98 requests per second
Can you give the mongo and reddis version numbers?
Just curious if you were using reddis 2.0 or possibly 2.2 ? Also what version of the reddis client were you using? There were some performance improvements recently.
Edit: thanks for sharing your experience, even if your test wasn't done with the help of a nobel winning computer team, it can still be useful data.
Just curious if you were using reddis 2.0 or possibly 2.2 ? Also what version of the reddis client were you using? There were some performance improvements recently.
Edit: thanks for sharing your experience, even if your test wasn't done with the help of a nobel winning computer team, it can still be useful data.
To know the version may be interesting but not really useful, I bet that if you change Redis with a dummy TCP server that is only able to reply +OK, whatever it receives, you'll get the same numbers here.
Basically every kind of networked server with a simple request-response protocol, in a busy loop over loopback, will show performances between 10k and 20k requests per second, mostly up to the speed of the implementation of the client.
Basically every kind of networked server with a simple request-response protocol, in a busy loop over loopback, will show performances between 10k and 20k requests per second, mostly up to the speed of the implementation of the client.
I was using redis 2.0 and mongo 1.6.5 with the latest drivers for each.
In his test, it appears the Mongo driver is just putting the message on the wire, not actually waiting on the server reply or verifying that the command was ever actually processed. Using the Redis in pipelined execution mode would be a much closer test.
My gut feeling tells me Redis should be faster at this, and as everyone else has pointed out, the methods used to arrive at these conclusions are most certainly questionable.
Redis is asynchronous, try doing it with a few more connections.
MongoDB is web scale.
I think that meme is funny too, but lets keep meme only posts over on reddit.
Sorry. I'll remember in the future that Hacker News is no place for humor.
Thanks! That's a fine executive summary. The longer version is that original humor is appreciated, particularly if insightful and non-obvious, but regurgitated memes are not welcome here. If you are rushing to make the obvious joke before everyone else does, please don't.
Seriously. I did not know that the use of a meme is violates site ethics guidelines. I checked the comment guidelines (http://ycombinator.com/newsguidelines.html) I didn't see any guideline referring to the use of memes. If using a meme is such an egregious offense as this comment thread suggests, maybe this violation should be mentioned in the guidelines.
Generally the popularity of a comment is a good enough indicator of how well a comment is received. If I got 100 upvotes on that comment I may get the impression that people like that. Considering I have < 0 on the comment, I can take the hint. I don't need the proper patrol reprimanding me about a non-stipulated rule.
Generally the popularity of a comment is a good enough indicator of how well a comment is received. If I got 100 upvotes on that comment I may get the impression that people like that. Considering I have < 0 on the comment, I can take the hint. I don't need the proper patrol reprimanding me about a non-stipulated rule.
Can you post the Redis configuration you are using?
of course none of it matters if the machine/vm crashes before the increments are committed to disk. :-)
For sure it seems like that MongoDB client library you are using has a smaller latency, perhaps it is a C library? As Redis-rb is instead written in Ruby itself.
Another huge problem of this benchmark is that it should try to increment a random key per every iteration, out of a dataset with a few millions of keys, as this is realistically what you need. You'll see how Redis with 50 clients will increment this counters 100k times per second or more without any performance degradation as the number of keys grows.