Introducing Disque [video](thedotpost.com)
thedotpost.com
Introducing Disque [video]
http://www.thedotpost.com/2015/06/salvatore-sanfilippo-introducing-disque
12 comments
For anyone curious, Disque is a synchronously replicated in-memory message broker/job queue that was designed to capture the essence of the "Redis as a jobs queue" use case.
Hi Antirez, always nice to use and see work from a good professional and outstanding community member. Whenever I saw people using Redis as a queue server, it was mainly for two reasons: 1) they were already using Redis and it had the right primitives, 2) it was dead simple to setup and use (compared to rabbitmq, for example).
One guy that I think is often overlooked is beanstalkd (https://github.com/kr/beanstalkd), especially with regards to 2). Have you ever taken a look at the project and if so, how do you think it compares to your vision for Disque?
One guy that I think is often overlooked is beanstalkd (https://github.com/kr/beanstalkd), especially with regards to 2). Have you ever taken a look at the project and if so, how do you think it compares to your vision for Disque?
As the author of one of the two systems I don't feel super well at comparing Disque and Benstalkd, but I guess, as Disque matures, people informed about both systems will compare them. However an obvious difference is that Disque is "internally" distributed and federated, while Benstalkd requires the client to take care of managing multiple nodes.
antirez as you mentioned both projects (Redis, Disque) are based on the same "framework" (skeleton).
Do you think because of their different guarantees that shared code will diverge?
Do you think that base layer will have to be kept in sync? (bug fixes, improvements)
I guess I'm asking, if at all, at what point would you pull that skeleton out into a separate repo and share it across Redis and Disque?
Do you think because of their different guarantees that shared code will diverge?
Do you think that base layer will have to be kept in sync? (bug fixes, improvements)
I guess I'm asking, if at all, at what point would you pull that skeleton out into a separate repo and share it across Redis and Disque?
Initially I had to face this choice, and I decided to go for the independent fork for a couple of reasons:
1) As you said, my guess was that the core part would diverge dramatically, if not for the separated files that can be kept in sync anyway, like event loop, string library, and so forth. And the time is showing this assumption was true. The code is diverging and as it diverges the effort to take it under the same library may become complex.
2) This was even important than "1": the effort to rewrite Redis in terms of a common core was a big one. I was not able to stop the development to undertake this effort, just for a side project which was not clear at the time where was headed. Now I'm more confident that Disque will get my attentions for the next years, but still I don't have the time to attempt such a big refactoring / rewrite.
However I cherry-pick from the common parts, and do some "diff" from time to time to see if there are ideas in one side that can be merged in the other side. Is not optimal, but I had no choices. What can be done in the future is to turn more parts of both systems into common libraries that can live in separated files, and rewrite only those parts, incrementally.
1) As you said, my guess was that the core part would diverge dramatically, if not for the separated files that can be kept in sync anyway, like event loop, string library, and so forth. And the time is showing this assumption was true. The code is diverging and as it diverges the effort to take it under the same library may become complex.
2) This was even important than "1": the effort to rewrite Redis in terms of a common core was a big one. I was not able to stop the development to undertake this effort, just for a side project which was not clear at the time where was headed. Now I'm more confident that Disque will get my attentions for the next years, but still I don't have the time to attempt such a big refactoring / rewrite.
However I cherry-pick from the common parts, and do some "diff" from time to time to see if there are ideas in one side that can be merged in the other side. Is not optimal, but I had no choices. What can be done in the future is to turn more parts of both systems into common libraries that can live in separated files, and rewrite only those parts, incrementally.
I find it interesting that the article claims eventual consistency, but also best effort ordering. The consistency in a queue is surely made of two parts: count and order. To not have duplicates is nice, but if ordering is not guaranteed - is this really a queue? Seems like it's a Something In Something Out, or "a pile" or something. Again, curious about what the evental consistency guarantee is referring to.
Not saying it cannot be useful - for work distribution it will surely work - but it needs to be considered for your usecase that you will need to be able to process messages in an unlimited unordering. Process a delete action before a create for example.
Not saying it cannot be useful - for work distribution it will surely work - but it needs to be considered for your usecase that you will need to be able to process messages in an unlimited unordering. Process a delete action before a create for example.
What is eventually consistent is the job state: it means that jobs are removed only when they are successfully delivered at least one time.
Best effort ordering does not mean no ordering btw. Use cases for which strict ordering is needed are rare, and it is possible to build order out of an unordered queue if strictly needed (especially since, when strict ordering is needed, usually you have a source of truth to maintain the state of the application, that happens to be a strictly consistent store).
The cost of ordering is huge in terms of design, and benefits just an handful of use cases. So I think best effort ordering is one of the best Disque assets.
Best effort ordering does not mean no ordering btw. Use cases for which strict ordering is needed are rare, and it is possible to build order out of an unordered queue if strictly needed (especially since, when strict ordering is needed, usually you have a source of truth to maintain the state of the application, that happens to be a strictly consistent store).
The cost of ordering is huge in terms of design, and benefits just an handful of use cases. So I think best effort ordering is one of the best Disque assets.
Ah OK. This might sound harsh, but so the guarantee you have is that the data will not be lost? That's a pretty weak guarantee no?
Also, best effort implies no ordering in the worst case no? I always try to judge systems when the world isn't peaches.
Also, best effort implies no ordering in the worst case no? I always try to judge systems when the world isn't peaches.
As someone who has to maintain a RabbitMQ cluster.. I cannot wait to be able to use this reliabily.
Does Disque support recurring jobs, like queue this task, and run it once every 4 hours. I know Antirez mentioned there is a delay property in the video.
Also will there be support for absolute scheduling like, run once at 4:30:00am today.
Also will there be support for absolute scheduling like, run once at 4:30:00am today.
$ ./antirez --voice "godfather" --talk
(I hope is understandable at least...)
(I hope is understandable at least...)
So nice to see you post this here. While watching I was really enjoying your Italian accent and was thinking about how amazing it is that so many people have to learn English. Reminded me of your post "English has been my pain for 15 years" (http://antirez.com/news/61). Thank you for conquering the language and presenting this to us. Really glad to have another message queue to explore.
Thanks linkmotif, when I finished talking, as usually, there were people literally ROTFLing for the accent, but in a positive way. Finally at least I can kinda talk in a more casual way, it used to be harder than this in the past because of totally lack of practice, usually I just write/read. Thanks for your message :-)
Developing oral skills in language learning out of context (i.e. some place where it is spoken) is a schizophrenic thing. I have somehow put on a british accent two years ago (mother tongue turkish), and the process was like this: watch lots of programmers' talks, and read out loud, being conscious as to my articulation, all the blog posts that I happen to read, and sometimes make monologues alone in my room. And my daily input was vast, as I was having a short two-year break from the uni. Now I'm studying Italian language and literature at the university, and am proving to myself that the way I learned english was spot on, self immersion is incredibly fruitful. One should merely be able to tolerate not understanding everything, written or spoken. And while you definitely have the italian accent, you are speaking is excellent, congratulations.
I am a native english speaker (born/raised in California) and also very self conscious about the way I sound for different reasons. I suspect many of us have this kind of internal battle, even if it seems relatively insignificant to other people.
FWIW, I think your accent is interesting and enjoyable to hear.
FWIW, I think your accent is interesting and enjoyable to hear.
Understood everything no problem. Having recently implemented a job queue system using Redis I am excited to see Disque.
I wonder if it makes sense to use Disque in the same environment as Redis, since we have use cases for both a key-value store as well as a job queue.
I wonder if it makes sense to use Disque in the same environment as Redis, since we have use cases for both a key-value store as well as a job queue.
Thanks, definitely makes sense to use Disque for messaging and Redis for caching/storage, and for messaging too when you need ephemeral messaging without guarantees, but you need to control very well the semantics, so in Redis you can build your messaging thing using its primitives. But for reliable message exchanges, with guaranteed at least once delivery, Disque is the thing to use (or other good & reliable message queues).
[deleted]
I've never had issues understanding you before man! Always amazing to hear your thoughts out loud because it shows how much you truly care about the projects you put so much time in :D
Looks interesting. But why would I use this instead of RabbitMQ ? (I didn't watch the video, but I did check the github repo)
I reckon it'll be a complexity thing.
Disque uses the same kind of tremendously simple API as Redis does. It appears to have ~15 commands, and is basically just a job broker and nothing else. If it gains traction, it will solve a specific common problem well.
RabbitMQ is a more comprehensive message broker, and more complex. It ships with federation, an HTTP API, SASL, that sort of thing. Useful tool, but likely more complex to setup and administer (to the point that it might be excessive for some use cases).
Obviously this is all subject to personal suitability though!
Disque uses the same kind of tremendously simple API as Redis does. It appears to have ~15 commands, and is basically just a job broker and nothing else. If it gains traction, it will solve a specific common problem well.
RabbitMQ is a more comprehensive message broker, and more complex. It ships with federation, an HTTP API, SASL, that sort of thing. Useful tool, but likely more complex to setup and administer (to the point that it might be excessive for some use cases).
Obviously this is all subject to personal suitability though!
How does this compare to Resque (I've used that in production and like it a lot)?
Resque is based on Redis, so it is similar but can't offer the same guarantees, scalability and fault tolerance. However for many use cases where ephemeral messaging is enough, Resque is a good pick.
I love Redis and I love messaging systems, so will definitely take a hard look at Disque, thanks for all your hard work!
I'm pretty excited – I've spent a couple of weeks on and off trying to build a useful reliable job queue system on top of Redis, but it's not really working as simply as I'd like.
Disque seems to offer the features I am interested in along with the relatively simple semantics of Redis. Looking forward to the API being settled a bit so I can play with it.
Disque seems to offer the features I am interested in along with the relatively simple semantics of Redis. Looking forward to the API being settled a bit so I can play with it.
Thank you for providing yet another project to the open source community Salvatore.
Thanks! My pleasure, and I'm already receiving tons of help from the community.