Python messaging with ActiveMQ and RabbitMQ(nighttale.net)

20 pointsby spidamanvor 17 Jahren7 comments

Python messaging with ActiveMQ and RabbitMQ

http://www.nighttale.net/activemq/python-messaging-activemq-and-rabbitmq.html

8 comments

anateusvor 17 Jahren

I would be interested to see a detailed comparison of the above and SQS which I've used to great success with the python library boto (http://code.google.com/p/boto/). And yes, I googled and haven't found anything good :)

I haven't used ActiveMQ before due to no reason other than horror stories from basically everyone I know who has used it heavily.

RabbitMQ seems like an interesting solution to explore if I ever need more speed than SQS provides.

spidamanvor 17 Jahren

Yea, boto is a great library. IME, SQS seems to be a good option only when you have low throughput. I tried running a high volume of jobs through SQS and found that I needed a lot more workers (EC2 instances) to do the work because each message was so slow to pickup. That made things cost prohibitive. I'm currently working on another AWS based project. It was initially using ActiveRecord based db queues but that, of course, failed at high volume. So I replaced it with kestrel, which works with any memcached client, and that's been great. However, kestrel doesn't do topic/group subscriptions where lots of receivers can get the same message. That's where these message servers seem to be potentially valuable options.

utku_karatas2vor 17 Jahren

Obtuse question - what are the practical use cases for these ..umm.. enterprisey solution things? What problem do they solve?

azimvor 17 Jahren

Very useful in a distributed systems like environment. Imagine process A needs to send some data to B, but either could reside anywhere in the cluster. Each just has to register with the broker and it does the dirty work of figuring out how to route messages.

gthankvor 17 Jahren

It's handy when certain actions that your system performs is really slow compared to everything else. You send a message to a queue that will result in the action eventually occurring, and then you go on with your business. One example would be building a full-text index of a large file (or group of files). You don't need to hold up your user while that slow process goes, so you send it to a queue and let worker processes do their thing. If the process is not only slow, but also compute-intensive, these worker processes can be on a different machine/farm of machines.

spidamanvor 17 Jahren

If you've never needed asynchronous systems and messaging, perhaps you should consider yourself lucky.

utku_karatas2vor 17 Jahren

Similar small scale problems of mine were easy to solve with Python Queue's. Maybe I am lucky those problems weren't at a scale azim and gthank explained though. oh the goose-bump :)

srwvor 17 Jahren