Head dev of RabbitMQ response to OpenAMQ's announcement(lists.rabbitmq.com)
lists.rabbitmq.com
Head dev of RabbitMQ response to OpenAMQ's announcement
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-March/006754.html
3 comments
Not much by way of technical content. Although when the original charge is that the protocol is "fundamentally flawed", I don't know how you can respond specifically beyond saying "it works and people are using it" (which it seems to and which they are).
You could say: "Yes, you are right. We, too, have noticed what huge pile of hopelessly overengineered garbage it is. Thanks for admitting your own mistake, so glad we can finally get rid of this monstrosity and move on to something sane."
This would be the automatic response from anyone who has had the misfortune to work with AMQP, thus I'm a bit baffled the RabbitMQ guys want to stick with it.
Edit: And yes, I'm not really surprised that I'm getting modded into oblivion. Still, would the downvoters mind to elaborate on what kind of exposure they've had to AMQP and what part of it they consider worth keeping?
This would be the automatic response from anyone who has had the misfortune to work with AMQP, thus I'm a bit baffled the RabbitMQ guys want to stick with it.
Edit: And yes, I'm not really surprised that I'm getting modded into oblivion. Still, would the downvoters mind to elaborate on what kind of exposure they've had to AMQP and what part of it they consider worth keeping?
Hi, Alexis here from RabbitMQ.
Wtf? You may not like what we do with AMQP but I can assure that lots of people are very happy using it in its RabbitMQ incarnation. Which version of AMQP have you "had the misfortune to work with" and how have you found it "hopelessly overengineered"? Is there something else you would recommend - and if so why?
We do actually like making our product better and appreciate constructive criticism.
Please do explain.
Also please note that RabbitMQ supports multiple protocols and one reason we like AMQP 0-8 and 0-9-1 is that they are flexible enough to make this easy (see e.g. RabbitHub on Github).
Cheers
alexis
Wtf? You may not like what we do with AMQP but I can assure that lots of people are very happy using it in its RabbitMQ incarnation. Which version of AMQP have you "had the misfortune to work with" and how have you found it "hopelessly overengineered"? Is there something else you would recommend - and if so why?
We do actually like making our product better and appreciate constructive criticism.
Please do explain.
Also please note that RabbitMQ supports multiple protocols and one reason we like AMQP 0-8 and 0-9-1 is that they are flexible enough to make this easy (see e.g. RabbitHub on Github).
Cheers
alexis
Oh interesting to hear from a RabbitMQ developer, thanks for taking the time to reply.
Well, I'll try to elaborate on my rant. Most of it from memory (been a year ago since I last touched AMQP).
Back then when I looked at AMQP (and used it for one project):
* It was very hard to find reasonable documentation. The most accessible I found was the (excellent) http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ blog post. This was what eventually got us off the ground. Imho a testament to the shortcomings of the "official" documentation (where is it?).
* RabbitMQ only implemented AMQP 0.8. I cannot remember the exact differences between 0.9 and 0.8 (and heck, good luck trying to find a document explaining them even today). For us it boiled down to some clients simply not working and it took a while to figure out what the problem even was.
* Client interoperability was generally spotty, even in clients supposed to be compatible. It seems the idea of that whole spec-file stuff was to make the protocol pluggable and flexible. However the only thing it seems to have achieved is to make client development hell. The client we eventually went with (py-amqplib) doesn't even use the spec-file because "parsing takes too long".
* Moreover the sheer existence of http://www.rabbitmq.com/interoperability.html is a testament to a flawed protocol. If I can't rely on a 0.8 client to talk to a 0.8 server properly, then something is fundamentally wrong.
* There's a huge pile of semantic baggage (Exchanges, Bindings, Queues, Channels) that left us with a giant "WTF" the more we used it. Yes, it can be made work. But hell, talk about overengineering. Much simpler constructions could have been used to achieve the same goals.
* As I just learned during my googling for this post, apparently some of the problems have even been acknowledged by the officials and are supposed to be simplified in AMQP 1.0 (http://www.amqp.org/confluence/display/AMQP/Background+to+AM...). But I'm not expecting much, to be honest.
So, that's just a few points that popped into my head immediately, there's more if I'd dig deeper. It's just a fundamentally bad protocol if I have ever seen one, and I'm honestly surprised to see people defend it even after the creators themselves (iMatix) acknowledge it to be a dead-end.
To put it from an different angle: When you think about what AMQP really does then how long would it take you to whip up a near-trivial but feature-equivalent protocol from scratch? I could think of many ways to skin that cat in a better way, with a simple, hackable, plain-text protocol.
You know, one of those protocols where I, as a developer, can even sniff and see what is going on at any time. Those are possible, as demonstrated by redis, memcached, STOMP, beanstalkd and many others.
I acknowledge that some complexity is unavoidable for advanced use-cases. Still, a good protocol layers the complexity in a way so that simple stuff remains simple and complex stuff becomes manageable. To me AMQP is a complete failure in both regards; you are confronted with the full complexity even for the most trivial use-case. And it seems for complex use-cases it adds a hefty layer of complexity of its own, rather than helping to wrap up.
Well, I'll try to elaborate on my rant. Most of it from memory (been a year ago since I last touched AMQP).
Back then when I looked at AMQP (and used it for one project):
* It was very hard to find reasonable documentation. The most accessible I found was the (excellent) http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ blog post. This was what eventually got us off the ground. Imho a testament to the shortcomings of the "official" documentation (where is it?).
* RabbitMQ only implemented AMQP 0.8. I cannot remember the exact differences between 0.9 and 0.8 (and heck, good luck trying to find a document explaining them even today). For us it boiled down to some clients simply not working and it took a while to figure out what the problem even was.
* Client interoperability was generally spotty, even in clients supposed to be compatible. It seems the idea of that whole spec-file stuff was to make the protocol pluggable and flexible. However the only thing it seems to have achieved is to make client development hell. The client we eventually went with (py-amqplib) doesn't even use the spec-file because "parsing takes too long".
* Moreover the sheer existence of http://www.rabbitmq.com/interoperability.html is a testament to a flawed protocol. If I can't rely on a 0.8 client to talk to a 0.8 server properly, then something is fundamentally wrong.
* There's a huge pile of semantic baggage (Exchanges, Bindings, Queues, Channels) that left us with a giant "WTF" the more we used it. Yes, it can be made work. But hell, talk about overengineering. Much simpler constructions could have been used to achieve the same goals.
* As I just learned during my googling for this post, apparently some of the problems have even been acknowledged by the officials and are supposed to be simplified in AMQP 1.0 (http://www.amqp.org/confluence/display/AMQP/Background+to+AM...). But I'm not expecting much, to be honest.
So, that's just a few points that popped into my head immediately, there's more if I'd dig deeper. It's just a fundamentally bad protocol if I have ever seen one, and I'm honestly surprised to see people defend it even after the creators themselves (iMatix) acknowledge it to be a dead-end.
To put it from an different angle: When you think about what AMQP really does then how long would it take you to whip up a near-trivial but feature-equivalent protocol from scratch? I could think of many ways to skin that cat in a better way, with a simple, hackable, plain-text protocol.
You know, one of those protocols where I, as a developer, can even sniff and see what is going on at any time. Those are possible, as demonstrated by redis, memcached, STOMP, beanstalkd and many others.
I acknowledge that some complexity is unavoidable for advanced use-cases. Still, a good protocol layers the complexity in a way so that simple stuff remains simple and complex stuff becomes manageable. To me AMQP is a complete failure in both regards; you are confronted with the full complexity even for the most trivial use-case. And it seems for complex use-cases it adds a hefty layer of complexity of its own, rather than helping to wrap up.
OK, here's my reply:
> Well, I'll try to elaborate on my rant. Most of it from memory (been a year ago since I last touched AMQP). Back then when I looked at AMQP (and used it for one project):
AMQP is quite new - think about the early days of HTTP and XMPP. Some of the things you list have improved in the last year - but we have more work to do.
> * It was very hard to find reasonable documentation. The most accessible I found was the (excellent) http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ blog post. This was what eventually got us off the ground.
It's a very good post. I think the documentation at least as far as RabbitMQ goes has improved, see the link store here: http://www.rabbitmq.com/how.html and the del.icio.us repo that it links to, which is organised by tag.
We'd very much appreciate assistance in improving all this!
> Imho a testament to the shortcomings of the "official" documentation (where is it?).
The working group has not done a good job at putting out 'official' documents about AMQP, other than the specs, which do provide documentation. For some reason people choose to not read the specs, but I do recommend the 0-9-1 spec which is about 40 pages long (not too bad considering).
There will be more of a push to do 'official' AMQP docs this year.
> * RabbitMQ only implemented AMQP 0.8. I cannot remember the exact differences between 0.9 and 0.8 (and heck, good luck trying to find a document explaining them even today).
We do actually have such a doc but have not released it since almost everyone uses 0-8. The main point is that 0-8 and 0-9 are almost identical.
> For us it boiled down to some clients simply not working and it took a while to figure out what the problem even was.
This has all got better, there are hundreds of projects now that work with RabbitMQ: http://delicious.com/alexisrichardson/rabbitmq+usecase+clien...
> * Client interoperability was generally spotty, even in clients supposed to be compatible. It seems the idea of that whole spec-file stuff was to make the protocol pluggable and flexible. However the only thing it seems to have achieved is to make client development hell. The client we eventually went with (py-amqplib) doesn't even use the spec-file because "parsing takes too long".
Most people have a good experience with most clients. I am sorry you did not - did you try asking for help on the mailing list?
> * Moreover the sheer existence of http://www.rabbitmq.com/interoperability.html is a testament to a flawed protocol.
Er.. no. It's testament to our desire to be clear about semantics.
You mention STOMP - I assume you are aware that STOMP's semantics are underspecified so that interoperability is not possible without vendor agreement?
> If I can't rely on a 0.8 client to talk to a 0.8 server properly, then something is fundamentally wrong.
How long did it take for HTTP browsers to work with all web servers? Years and years - and most people only use a subset of HTTP.
> * There's a huge pile of semantic baggage (Exchanges, Bindings, Queues, Channels) that left us with a giant "WTF" the more we used it. Yes, it can be made work. But hell, talk about overengineering. Much simpler constructions could have been used to achieve the same goals.
There are a lot of messaging protocols, and none of them have managed to be complete, interoperable, open and simple. Please do help us to make RabbitMQ better, whether for AMQP or one of the other protocols it supports.
> * As I just learned during my googling for this post, apparently some of the problems have even been acknowledged by the officials and are supposed to be simplified in AMQP 1.0 (http://www.amqp.org/confluence/display/AMQP/Background+to+AM...).
Those comments refer to 0-10 which is a lot more complex.
> But I'm not expecting much, to be honest.
That's encouraging :-(
Why not? You can help with this stuff you know.
> So, that's just a few points that popped into my head immediately, there's more if I'd dig deeper. It's just a fundamentally bad protocol if I have ever seen one,
Thanks for taking the time to provide feedback. I'm not sure what to conclude from your specific comments. The interop stuff has been cleaned up in 0-9-1. The fact that people get confused by Exchanges vs Queues, and the programmable nature of AMQP, is being addressed in 1-0.
In any case, as far as RabbitMQ goes, it's a multiprotocol broker and so far we have learnt two things about using AMQP - (1) that it's a good core protocol for messaging, around which other gateways are easily provided; (2) yes the API could be simpler - we are working on that.
> and I'm honestly surprised to see people defend it even after the creators themselves (iMatix) acknowledge it to be a dead-end.
I think you are reading a lot into iMatix's public statements that may not be borne out by the facts. You may find this odd, for example: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-Ma...
> To put it from an different angle: When you think about what AMQP really does then how long would it take you to whip up a near-trivial but feature-equivalent protocol from scratch? I could think of many ways to skin that cat in a better way, with a simple, hackable, plain-text protocol.
I agree that plain text would be good, but it does not cut it for some cases. I'm a fan of plain text nonetheless. But if you think this is so easy, please get involved and help!
> You know, one of those protocols where I, as a developer, can even sniff and see what is going on at any time. Those are possible, as demonstrated by redis, memcached, STOMP, beanstalkd and many others.
I agree that plain text has some advantages, and am a huge fan of Redis, but don't see any of those as delivering a messaging protocol. This stuff is non-trivial unfortunately; as I say we'd welcome your input.
> I acknowledge that some complexity is unavoidable for advanced use-cases. Still, a good protocol layers the complexity in a way so that simple stuff remains simple and complex stuff becomes manageable. To me AMQP is a complete failure in both regards; you are confronted with the full complexity even for the most trivial use-case. And it seems for complex use-cases it adds a hefty layer of complexity of its own, rather than helping to wrap up.
I definitely agree that most users want a simple entry point.
Cheers,
alexis
> Well, I'll try to elaborate on my rant. Most of it from memory (been a year ago since I last touched AMQP). Back then when I looked at AMQP (and used it for one project):
AMQP is quite new - think about the early days of HTTP and XMPP. Some of the things you list have improved in the last year - but we have more work to do.
> * It was very hard to find reasonable documentation. The most accessible I found was the (excellent) http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ blog post. This was what eventually got us off the ground.
It's a very good post. I think the documentation at least as far as RabbitMQ goes has improved, see the link store here: http://www.rabbitmq.com/how.html and the del.icio.us repo that it links to, which is organised by tag.
We'd very much appreciate assistance in improving all this!
> Imho a testament to the shortcomings of the "official" documentation (where is it?).
The working group has not done a good job at putting out 'official' documents about AMQP, other than the specs, which do provide documentation. For some reason people choose to not read the specs, but I do recommend the 0-9-1 spec which is about 40 pages long (not too bad considering).
There will be more of a push to do 'official' AMQP docs this year.
> * RabbitMQ only implemented AMQP 0.8. I cannot remember the exact differences between 0.9 and 0.8 (and heck, good luck trying to find a document explaining them even today).
We do actually have such a doc but have not released it since almost everyone uses 0-8. The main point is that 0-8 and 0-9 are almost identical.
> For us it boiled down to some clients simply not working and it took a while to figure out what the problem even was.
This has all got better, there are hundreds of projects now that work with RabbitMQ: http://delicious.com/alexisrichardson/rabbitmq+usecase+clien...
> * Client interoperability was generally spotty, even in clients supposed to be compatible. It seems the idea of that whole spec-file stuff was to make the protocol pluggable and flexible. However the only thing it seems to have achieved is to make client development hell. The client we eventually went with (py-amqplib) doesn't even use the spec-file because "parsing takes too long".
Most people have a good experience with most clients. I am sorry you did not - did you try asking for help on the mailing list?
> * Moreover the sheer existence of http://www.rabbitmq.com/interoperability.html is a testament to a flawed protocol.
Er.. no. It's testament to our desire to be clear about semantics.
You mention STOMP - I assume you are aware that STOMP's semantics are underspecified so that interoperability is not possible without vendor agreement?
> If I can't rely on a 0.8 client to talk to a 0.8 server properly, then something is fundamentally wrong.
How long did it take for HTTP browsers to work with all web servers? Years and years - and most people only use a subset of HTTP.
> * There's a huge pile of semantic baggage (Exchanges, Bindings, Queues, Channels) that left us with a giant "WTF" the more we used it. Yes, it can be made work. But hell, talk about overengineering. Much simpler constructions could have been used to achieve the same goals.
There are a lot of messaging protocols, and none of them have managed to be complete, interoperable, open and simple. Please do help us to make RabbitMQ better, whether for AMQP or one of the other protocols it supports.
> * As I just learned during my googling for this post, apparently some of the problems have even been acknowledged by the officials and are supposed to be simplified in AMQP 1.0 (http://www.amqp.org/confluence/display/AMQP/Background+to+AM...).
Those comments refer to 0-10 which is a lot more complex.
> But I'm not expecting much, to be honest.
That's encouraging :-(
Why not? You can help with this stuff you know.
> So, that's just a few points that popped into my head immediately, there's more if I'd dig deeper. It's just a fundamentally bad protocol if I have ever seen one,
Thanks for taking the time to provide feedback. I'm not sure what to conclude from your specific comments. The interop stuff has been cleaned up in 0-9-1. The fact that people get confused by Exchanges vs Queues, and the programmable nature of AMQP, is being addressed in 1-0.
In any case, as far as RabbitMQ goes, it's a multiprotocol broker and so far we have learnt two things about using AMQP - (1) that it's a good core protocol for messaging, around which other gateways are easily provided; (2) yes the API could be simpler - we are working on that.
> and I'm honestly surprised to see people defend it even after the creators themselves (iMatix) acknowledge it to be a dead-end.
I think you are reading a lot into iMatix's public statements that may not be borne out by the facts. You may find this odd, for example: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-Ma...
> To put it from an different angle: When you think about what AMQP really does then how long would it take you to whip up a near-trivial but feature-equivalent protocol from scratch? I could think of many ways to skin that cat in a better way, with a simple, hackable, plain-text protocol.
I agree that plain text would be good, but it does not cut it for some cases. I'm a fan of plain text nonetheless. But if you think this is so easy, please get involved and help!
> You know, one of those protocols where I, as a developer, can even sniff and see what is going on at any time. Those are possible, as demonstrated by redis, memcached, STOMP, beanstalkd and many others.
I agree that plain text has some advantages, and am a huge fan of Redis, but don't see any of those as delivering a messaging protocol. This stuff is non-trivial unfortunately; as I say we'd welcome your input.
> I acknowledge that some complexity is unavoidable for advanced use-cases. Still, a good protocol layers the complexity in a way so that simple stuff remains simple and complex stuff becomes manageable. To me AMQP is a complete failure in both regards; you are confronted with the full complexity even for the most trivial use-case. And it seems for complex use-cases it adds a hefty layer of complexity of its own, rather than helping to wrap up.
I definitely agree that most users want a simple entry point.
Cheers,
alexis
> how long would it take you to whip up a near-trivial but feature-equivalent protocol from scratch?
Oh, about five years, I'd say.
Oh, about five years, I'd say.
Hey thanks for that. Good response. I'm travelling so a full response is on its way soon..ish.
I'm not down-modding you, but I am using AMQP in production on http://tweeteorites.com/ with RabbitMQ.
I need persistence and flow control, as my consumers are unreliable and production rate is highly variable. This will become even more important as some of the new Twitter data streams come online.
It took me about 2 days to completely reengineer my backend around RabbitMQ and a custom daemon to serve stream requests. Granted, I already had RabbitMQ set up and routing background tasks for celery ( http://celeryproject.org/ ), but it's still pretty trivial to set up and get running. The easy case, which I'm at now, is pretty easy, and I've got a good path to scalability with fan out exchanges.
That said, faster is better, and I'd be interested in zeromq if they had some of the features I need.
I need persistence and flow control, as my consumers are unreliable and production rate is highly variable. This will become even more important as some of the new Twitter data streams come online.
It took me about 2 days to completely reengineer my backend around RabbitMQ and a custom daemon to serve stream requests. Granted, I already had RabbitMQ set up and routing background tasks for celery ( http://celeryproject.org/ ), but it's still pretty trivial to set up and get running. The easy case, which I'm at now, is pretty easy, and I've got a good path to scalability with fan out exchanges.
That said, faster is better, and I'd be interested in zeromq if they had some of the features I need.
That is actually a very valid question - anyone know the stack similar to rabbitmq server + celeryd , but built around zeromq ?
Since ZeroMQ has no broker but is peer to peer, that would be somewhat unnatural.
Cheers
alexis
Cheers
alexis
[deleted]
ok, i am a bit clueless here, but if 0mq has no persistance, how is it reliable (durable)? or, if it's not reliable (durable), isn't this whole debate comparing apples and oranges?
[edited to clarify i am talking about durability and not just reliability in the tcp sense]
[edited to clarify i am talking about durability and not just reliability in the tcp sense]
You are correct, there is no persistence. 0MQ is basically just a step above socket level communications, while obviating all of the tedious BS socket programming forces you to do.
Here's a good intro to 0MQ from lwn.net, along with sample code: http://lwn.net/Articles/370307/
Here's a good intro to 0MQ from lwn.net, along with sample code: http://lwn.net/Articles/370307/
thanks. this seems an awful lot like the nosql "debate" in that the two "sides" (typically) have different priorities (and the differences are similar, related to consistency and scaling).
[and also makes imatix not look that good, if any customers need the durability part, which is quite a common requirement]
[and also makes imatix not look that good, if any customers need the durability part, which is quite a common requirement]
It is reliable in the sense that it can deliver messages between two peers reliably, like TCP. What you're speaking of is durability.
This doesn't address 0MQ's performance improvement claims, which I would have thought was the biggest thing the RabbitMQ guys would want to address.
It turns out that RabbitMQ's throughput of tens of thousands of transient messages per second per CPU is enough for a lot of cases. Similar story with latency.
ZeroMQ does a lot of neat stuff to make it go faster still, e.g taking out the server, and removing all routing info from the messages, but ends up being much more like an alternative to sockets as a result, which makes it less useful for general work.
In summary - RabbitMQ and ZeroMQ are complementary, rather than really being alternatives to one another.
alexis
RabbitMQ
ZeroMQ does a lot of neat stuff to make it go faster still, e.g taking out the server, and removing all routing info from the messages, but ends up being much more like an alternative to sockets as a result, which makes it less useful for general work.
In summary - RabbitMQ and ZeroMQ are complementary, rather than really being alternatives to one another.
alexis
RabbitMQ
If you don't need to provide persistence and flow control it is amazing how much faster you can go...
And if you do, things can became pretty complicated.
Absolutely. I happen to need these bits so zeromq is a non-starter for me. I am mildly annoyed by the "we are faster" rhetoric without the "but we only support a small subset of the useful things you can do with a message queue" caveat, but it is a mostly meaningless bit of marketing so I can let it slide; anyone who really needs those sorts of speeds knows how to actually evaluate products and is unlikely to be suckered by a false dichotomy setup like the one on the zeromq home page.