NSQ is a full-featured messaging platform out-of-the-box, whereas zeromq and nanomsq are lower level libraries that you could use to build (the same) functionality.
That sentence is missing an important word. No centralized brokers, meaning the typical/recommended deployment topology is an nsqd node on all hosts producing messages.
I appreciate all the work that Tyler has put into his series of blog posts on messaging systems (and NSQ bug reports!) but I think this particular article is one that misses the mark [1].
While raw performance is important, I think comparing the guarantees and related operational and development semantics are far more interesting and useful.
These systems vary a great deal in the interfaces and functionality they expose to users building on top of them and operators dealing with them when they're deployed in production (and inevitably break!). This is ultimately what matters most.
P.S. single node (localhost) performance is essentially useless. Most of these tools are designed to be deployed as the backbone of large distributed systems (NSQ in particular), so scalability and performance in that context is a better indication of real-world performance.
NSQ is as much about what it doesn't do as it is what it does. To a certain extent this mirrors, and was inspired by, the language's philosophy (Go) [1].
Also, NSQ was designed to replace an existing home-grown system deployed at scale. This dictated a lot of the initial requirements (and in certain cases excluded off-the-shelf tools).
When we left the experimental phase we realized we had built something that was useful to others, and it turns out that despite not having the features you've identified it can be incredibly effective in lots of use cases that don't need stronger guarantees.
[1] If I'm being honest, NSQ was a vehicle for adoption of Go at bitly as well as the project we used to learn the language. This was a huge risk at the time (almost 3 years ago) but one that has certainly paid off.
NSQ treats the message data as an opaque blob so the format wouldn't directly affect it (except on some lower level related to overall message size I suppose). It would impact your producers (encoding) and consumers (decoding), obviously.
For the record... NSQ is a generic messaging platform and is data format agnostic, there is nothing specific about metrics collection (it was just my contrived example).
Agreed. Message queues play an important role for us (bitly) in being a layer of fault tolerance, buffering, and a means to perform various operational tasks.
They're so important to us that we decided to build something that worked exactly like we wanted.
NSQ is a realtime distributed message processing system where we've taken the approach of focussing on making it ops friendly and easy to get started.
IMO, solutions that make it easy to develop on and administrate are most important... because things break. NSQ is straightforward to deploy (limit dependencies), simple to configure (runtime discovery), and client libraries provide a lot of functionality important for handling failures (like backoff, deferred messages, etc.) for a variety of use cases.
We've written an in-depth introductory blog post [2] about NSQ that has more details.
The focal point is discovery. Not other queues (or other libraries that can build queues). This is an interesting way to approach it, thanks for open sourcing.
We chose to solve the discovery problem a bit differently in NSQ (https://github.com/bitly/nsq) but I could certainly see some interesting opportunities to experiment with a distribute hash table approach as well.
The truth is we wanted to use ZeroMQ initially. It would have been really awesome to have all that flexibility on the client side.
The ZeroMQ documentation is fantastic as well. It inspired and helped shape some of the design choices we made.
The further along we got from design to implementation it became obvious that it would be important to "own" the socket. Generally speaking, this is exactly what ZeroMQ prevents you from doing (and rightly so, it aims to abstract all of that away).
The choice to use Go had an impact here as well. Language features like channels and the breadth of the standard library made it really easy to translate our NSQ design into working code, offsetting the benefit of ZeroMQ's abstractions.
That's fair, you certainly can use rabbitmq in a distributed and decentralized fashion.
You bring up an important point though... What is important is the topology we're promoting and the platform we've built to support it. The actual queue is less important and it made sense for us to own that piece to achieve our goals.
We looked into AMQP based solutions. Our understanding is that slaving, master-slave failover, or other strategies are used to mitigate the fact that there is a broker responsible for a given stream of messages.
We wanted a brokerless, decentralized, and distributed system. NSQ has no broker or SPOF so it is able to route around failures.
That said, I think RabbitMQ is a good tool depending on your requirements. I can imagine a broker proving useful in situations where you may want strict ordering or no-duplication. Those were tradeoffs we were willing to make for fault tolerance and availability.
Also, given the fact that we were already operating infrastructure built on simplequeue (which is also distributed and brokerless) we found it more appealing to evolve rather than replace.
beanstalkd is a great project and it certainly inspired some of our thinking. I think of beanstalkd as a more fully functional alternative to simplequeue (what we built/used prior to NSQ - more details in our blog post http://word.bitly.com/post/33232969144/nsq).
Some of the goals of NSQ transcended just replacing our specific daemon that buffered and delivered messages (most importantly the interactions with the lookup service). Because of that, we felt that owning that piece would make it easier to achieve those goals.
Additionally, one of the most important properties of nsqd (the queue component of NSQ) is that data is pushed to the client rather than polled (like in beanstalkd).