Evaluating persistent, replicated message queues(warski.org)
warski.org
Evaluating persistent, replicated message queues
http://www.warski.org/blog/2014/07/evaluating-persistent-replicated-message-queues/
5 comments
I am not very familiar with messaging or message queues, so wanted to seek out some advice before spending time reinventing some well known wheel poorly.
I would like to have a broker that sits between an application and several endpoint services to which the application makes requests via the broker. These endpoint services have rules, typically throttling rules (i) you cannot submit more than x number of request per second (ii) you cannot queue up more than y number of requests at our end (iii) you must put a delay of z seconds after we have responded to a request of yours, etc., etc.
Is there a defacto standard open source software for this situation, that would take a config file describing the policies to follow for an end point. My requirement is pretty toy'ish, small to medium scale only. Just taking a look at Redis docs (am new to that too) it seems it will not be that hard to write out a simple solution. Would that be the way to go ? Or is there something out there that already handles this.
I would like to have a broker that sits between an application and several endpoint services to which the application makes requests via the broker. These endpoint services have rules, typically throttling rules (i) you cannot submit more than x number of request per second (ii) you cannot queue up more than y number of requests at our end (iii) you must put a delay of z seconds after we have responded to a request of yours, etc., etc.
Is there a defacto standard open source software for this situation, that would take a config file describing the policies to follow for an end point. My requirement is pretty toy'ish, small to medium scale only. Just taking a look at Redis docs (am new to that too) it seems it will not be that hard to write out a simple solution. Would that be the way to go ? Or is there something out there that already handles this.
HornetQ has flow control (http://docs.jboss.org/hornetq/2.4.0.Final/docs/user-manual/h...). Not sure about e.g. Rabbit.
However I'm not aware of a standard solution in that space.
Take a look at Reactive Streams (http://www.reactive-streams.org/) as well.
However I'm not aware of a standard solution in that space.
Take a look at Reactive Streams (http://www.reactive-streams.org/) as well.
[deleted]
What this article fails to talk about are primarily the failure scenarios of the queues under partitions. Does the queue become write-unavailable, read-unavailable, inconsistent, or otherwise during network partition? Can the queue remain partially available during network partition?
These are the situations, in a scaling, production system that really matter.
These are the situations, in a scaling, production system that really matter.
These are very good questions, however for a different post. Here I was mainly interested in the throughput.
Jepsen post provide some insights (into Mongo and Rabbit). With Hornet you can easily end up with two primaries. SQS is untestable in that respect :).
Jepsen post provide some insights (into Mongo and Rabbit). With Hornet you can easily end up with two primaries. SQS is untestable in that respect :).
required reading for anyone using distributed systems: http://aphyr.com/tags/Jepsen
in particular:
http://aphyr.com/posts/284-call-me-maybe-mongodb
http://aphyr.com/posts/315-call-me-maybe-rabbitmq
in particular:
http://aphyr.com/posts/284-call-me-maybe-mongodb
http://aphyr.com/posts/315-call-me-maybe-rabbitmq
Would love to see Kafka represented here.
Yes, my bad, should be there :)
Stay tuned, I'm out soon till end of next week, but will definitely add it.
Stay tuned, I'm out soon till end of next week, but will definitely add it.
http://activemq.apache.org/
It's long been very popular, and has all the flexibility and enterprisey goodness you could want, but has not had a reputation for speed. However, i believe there was a release a year or two ago that significantly improved performance. I vaguely remember this being based on an infusion of technology from the Qpid project, which itself has a broker:
http://qpid.apache.org/components/java-broker/index.html
Or maybe that was what led to Apollo, which is the faster, more reliable successor to ActiveMQ, but which hasn't, er, succeeded ActiveMQ:
http://activemq.apache.org/apollo/
Or maybe i just imagined the link to Qpid.
I'm fairly hazy on how these bits of software relate, and i can't promise you there aren't more somewhere in the vast Apache mansion. Even figuring that out would be useful, never mind the benchmarks!