Building a Node.js Events App Using RabbitMQ, Websockets, and Django(jamesburkhart.com)
jamesburkhart.com
Building a Node.js Events App Using RabbitMQ, Websockets, and Django
http://jamesburkhart.com/building-a-nodejs-events-app-using-rabbitmq-w
6 comments
Thanks for the response! We went with Node.js instead of connecting the client directly to Rabbit primarily because of socket.io.
So far the amount of activity has been extremely underwhelming for the node instances, but we'll see if this changes over time. The current total activity amounts to <100 activities per minute, and I'm pretty sure we could support at least an order of magnitude more than that. Each node box has identical state at all times to all of the others, with the exception of currently connected socket.io clients.
Honestly the first thing I would do if we had the need to scale horizontally (scaling only for HA for now) would be to reduce the amount of data sent over the wire at all levels. I'd modify the current solution of sending the entire object to send only a reference to the object and what amounts to a diff of old version to new version.
I'm pretty new at architecting stuff like this (this project was my first time making architecture decisions), so I'm sure there's plenty of room for optimization. I'm just hoping that it was architected in a way such that if/when it fails due to load, very little of the code will need to be rewritten. However, given current performance, the increase in amount of activity that would be required to overload our architecture seems like one of those 'Good Problems to Have'.
So far the amount of activity has been extremely underwhelming for the node instances, but we'll see if this changes over time. The current total activity amounts to <100 activities per minute, and I'm pretty sure we could support at least an order of magnitude more than that. Each node box has identical state at all times to all of the others, with the exception of currently connected socket.io clients.
Honestly the first thing I would do if we had the need to scale horizontally (scaling only for HA for now) would be to reduce the amount of data sent over the wire at all levels. I'd modify the current solution of sending the entire object to send only a reference to the object and what amounts to a diff of old version to new version.
I'm pretty new at architecting stuff like this (this project was my first time making architecture decisions), so I'm sure there's plenty of room for optimization. I'm just hoping that it was architected in a way such that if/when it fails due to load, very little of the code will need to be rewritten. However, given current performance, the increase in amount of activity that would be required to overload our architecture seems like one of those 'Good Problems to Have'.
If you're a Django shop and only using node for socket.io, I recently released django-socketio which you'd probably find interesting: https://github.com/stephenmcd/django-socketio
Our thoughts on sockets -- SockJS
http://www.rabbitmq.com/blog/2011/09/13/sockjs-websocket-emu...
http://www.rabbitmq.com/blog/2011/09/13/sockjs-websocket-emu...
I notice there's a load-balancer in front of the nodejs servers. Would that become the single point of failure and single point of bottleneck? How many concurrent connections can the load-balancer take?
I asked on the blog, but since there's other people here who might be using these technologies: Why Node and WebSockets over Twisted and Athena/Minerva?
Did you consider skipping Nodejs and just rigging up a way for the browsers to subscribe directly to RabbitMQ over HTTP? We looked at that possibility but went with node (a) to make authorization code easier to write and (b) because of the availability socket.io as an easy abstraction over different socket-like techniques in the browser.
Also, are you worried that the node instances will get overwhelmed if they're receiving all activity through a fanout exchange? It sounds like each node instance will be responsible for figuring out how to route each piece of information coming through the system, instead of leaving this up to the RabbitMQ cluster.
Cheers though, looks like a fun project!