RSMQ: A lightweight Redis based message queue for Node.js(smrchy.github.io)
smrchy.github.io
RSMQ: A lightweight Redis based message queue for Node.js
http://smrchy.github.io/rsmq/
6 comments
MQTT uses a 2PC-like mechanism for exactly-once delivery [1]. The sender sends and the receiver acks as usual. But the message is only delivered once the sender then sends a release message, which the receiver acks. End result: sender can assume receiver distributed the message only once.
[1] http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-o...
[1] http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-o...
That's not exactly once. There are several failure modes that can cause duplication. You can detect duplication but not prevent it.
It depends on how you define the scope of the delivery, which is mentioned in the article. MQTT's exactly-once ensures that the receiver delivers the message to the end application exactly once, even if there had been duplicated messages between the sender and the receiver.
Sure. But that is true of all messaging systems. When people are discussing whether something is at-most-once or at-least-once they are talking about independent of the protocol.
The reason that is important is because it has large ramifications for other parts of the delivery promise (such as order guarantees) or the implementation of the protocol (such as how chatty MQTT is).
It is relatively trivial to implement a system with exactly once delivery if you allow for client filtering and don't have performance or delivery time constraints.
The reason that is important is because it has large ramifications for other parts of the delivery promise (such as order guarantees) or the implementation of the protocol (such as how chatty MQTT is).
It is relatively trivial to implement a system with exactly once delivery if you allow for client filtering and don't have performance or delivery time constraints.
[deleted]
It's not exactly "once". It's exactly "one" within a defined period of time. So in that articles context it would be the "at-least-once" variant.
If the recipient fails to process the message for some reason it will reappear after the visibility timeout.
If the recipient fails to process the message for some reason it will reappear after the visibility timeout.
I don't think you can provide exactly once even with that constraint. What if the defined period of time was 100 years?
Then you need a very big buffer.
First off, this looks great! It has a simple interface and could compliment some simple projects very well.
However, this quote, "If you run a Redis server and currently use Amazon SQS or a similar message queue you might as well use this fast little replacement" is greatly over simplifying things.
Amazon SQS gives you acknowledgement guarantees that are not possible with Redis. Don't get me wrong, I love Redis, but you can't reliably trust that writes happen.
https://aphyr.com/posts/283-call-me-maybe-redis
However, this quote, "If you run a Redis server and currently use Amazon SQS or a similar message queue you might as well use this fast little replacement" is greatly over simplifying things.
Amazon SQS gives you acknowledgement guarantees that are not possible with Redis. Don't get me wrong, I love Redis, but you can't reliably trust that writes happen.
https://aphyr.com/posts/283-call-me-maybe-redis
Nice library.
This is unrelated, but what are you using for the docs? I love how it looks and scrolls with the topics and code examples right beside the methods.
This is unrelated, but what are you using for the docs? I love how it looks and scrolls with the topics and code examples right beside the methods.
Shame they're unviewable with the HTTPS Everywhere plugin enabled.
Simply queues in Lua https://github.com/tarantool/queue i don't know what's reason to use redis+node.js, if exist tarantool queues
Did anybody here use the corresponding worker module too? How does it compare with kue (in production)?
We're have been using this in production for a few months now. We never tried kue but used the rsmq-worker module extensively.
Currently we're processing about 30 million messages per day on a single redis server with two worker instances.
Currently we're processing about 30 million messages per day on a single redis server with two worker instances.
Thanks, I will give it a try - even though I have been very happy with kue.
What does this do that I can't get from plain Redis Pub Sub?
This is a lot more than pub/sub.
A message queue stores messages so that nothing gets lost, when there are no subscribers listening.
A new message will be delivered to only one recipient at a time while in a pub/sub system all subscribers would receive a message and would either all work on the same message/job or would have to decide who does what.
Whoever receives a message will "work" on that message and after success will delete the message. Usually within seconds. So no one will ever receive that message again. If a receiving process crashes or some error happens the message will just pop up again and will be received again after a set invisibility time (default is 30s).
So dropping messages in this message queue keeps them there until some receiver(s) delete them. It does not matter how many message producers / consumers there are. The queue handles the problem of not delivering a message to two consumers within a set time.
A message queue stores messages so that nothing gets lost, when there are no subscribers listening.
A new message will be delivered to only one recipient at a time while in a pub/sub system all subscribers would receive a message and would either all work on the same message/job or would have to decide who does what.
Whoever receives a message will "work" on that message and after success will delete the message. Usually within seconds. So no one will ever receive that message again. If a receiving process crashes or some error happens the message will just pop up again and will be received again after a set invisibility time (default is 30s).
So dropping messages in this message queue keeps them there until some receiver(s) delete them. It does not matter how many message producers / consumers there are. The queue handles the problem of not delivering a message to two consumers within a set time.
I'll just leave this here for you: http://bravenewgeek.com/you-cannot-have-exactly-once-deliver...
Welcome to message queues. Where the network is both your best friend and enemy.