Resque Bus(tech.taskrabbit.com)
tech.taskrabbit.com
Resque Bus
http://tech.taskrabbit.com/blog/2013/09/28/resque-bus/
7 comments
>Currently all redis-based background job implementations does not support scheduled jobs.
That has been false for 4 years.
https://github.com/resque/resque-scheduler
That has been false for 4 years.
https://github.com/resque/resque-scheduler
It's polling based approach.
https://github.com/resque/resque-scheduler/blob/master/lib/r...
https://github.com/resque/resque-scheduler/blob/master/lib/r...
Resque will scan for set value with key prefix like "timestamps:" or "delayed:", for every five seconds, if there's any task queued it will be fired.
There's many pros/cons with this approach, e.g. in Redis SADD, SMEMBERS are O(N) operations while in theory dedicated cron algorithm could make O(log(n)).
https://en.wikipedia.org/wiki/Cron#Multi-user_capability
This is is important if you have many short lived, timestamp based events need to handle. My use case is design item cool down for a mobile/web game using server side push, with millions of users. While resque could in theory solve this problem, but it's no different from a per 5 minute crontab command.
So anyway, what I hope is a native clock in Redis handling all this.
https://github.com/resque/resque-scheduler/blob/master/lib/r...
https://github.com/resque/resque-scheduler/blob/master/lib/r...
Resque will scan for set value with key prefix like "timestamps:" or "delayed:", for every five seconds, if there's any task queued it will be fired.
There's many pros/cons with this approach, e.g. in Redis SADD, SMEMBERS are O(N) operations while in theory dedicated cron algorithm could make O(log(n)).
https://en.wikipedia.org/wiki/Cron#Multi-user_capability
This is is important if you have many short lived, timestamp based events need to handle. My use case is design item cool down for a mobile/web game using server side push, with millions of users. While resque could in theory solve this problem, but it's no different from a per 5 minute crontab command.
So anyway, what I hope is a native clock in Redis handling all this.
I know it is not quite what you are looking for, but if you looking at that type of cronjob replacement, did you not see a HN link to AirBNB guys talking about their tool Chronos? You might like that.
http://nerds.airbnb.com/introducing-chronos/
http://nerds.airbnb.com/introducing-chronos/
Sidekiq supports scheduled jobs: https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs
I preferred the approach described here over pure Redis pub/sub because of the ability to disconnect or lose connections and still end up with the data eventually.
Specifically about "cron" capability, resque-scheduler does a pretty good job for Resque in general. Resque Bus uses it for exponential-backoff retry.
It uses the scheduling capability for the "Heartbeat" described in the README. I'll write more about that later if people are interested. Quickly, it's an automated way to add events every minute. We use this like a global (shared across applications and servers) cron clock.
Specifically about "cron" capability, resque-scheduler does a pretty good job for Resque in general. Resque Bus uses it for exponential-backoff retry.
It uses the scheduling capability for the "Heartbeat" described in the README. I'll write more about that later if people are interested. Quickly, it's an automated way to add events every minute. We use this like a global (shared across applications and servers) cron clock.
If you don't mind running development versions: Notifications can be a part of building that. Problem: you (currently) have to be connected when it happens.
http://redis.io/topics/notifications
http://redis.io/topics/notifications
How do you handle exceptions and errors who would require a rework of the job?
That is interesting, I was looking for something similar to an EBS for PHP time ago, but eventually I wrote myself an ad hoc solution which uses MySQL (and row locks) plus a gearman worker and cron.
That is interesting, I was looking for something similar to an EBS for PHP time ago, but eventually I wrote myself an ad hoc solution which uses MySQL (and row locks) plus a gearman worker and cron.
Resque Bus uses resque-retry and resque-scheduler to do exponential backoff. That is, it will retry immediately, 2 seconds later, 4 seconds later, and so on up to several hours.
It was a little tricky to make sure it ended back in the right application queue, but it worked. If it was put in another one, it would crash because the code to process it would not line up.
This is truly awesome. I have been wanting this semanticto supplement my queues for a long time and was dressing adopting an operationally more complicated tool like Kafka, not to mention all the effort that would be needed to port code.
Why not use AMQP?
Edited a comment from the blog comments a bit...
I tried my best to explain some of the reasons we went this way instead of using things out there. I'm not claiming to have invented anything particularly novel here because it's a simple problem to move things from one list to N lists based on keys. We found value in doing it with technology that was already part of our stack and very well known amongst the team and community.
I did an implementation in RabbitMQ at the outset. I found that the simplicity of just moving the data around, clarity of persistence, and control over subscriptions was worth the relatively short time it took to implement this gem. For example, we have found value in matching on any of the or multiple published keys (specific values, regex, blank, present, not present, etc) as part of the subscription model. I could not figure this out using the routing_key and/or topic kind of stuff in RabbitMQ.
I tried my best to explain some of the reasons we went this way instead of using things out there. I'm not claiming to have invented anything particularly novel here because it's a simple problem to move things from one list to N lists based on keys. We found value in doing it with technology that was already part of our stack and very well known amongst the team and community.
I did an implementation in RabbitMQ at the outset. I found that the simplicity of just moving the data around, clarity of persistence, and control over subscriptions was worth the relatively short time it took to implement this gem. For example, we have found value in matching on any of the or multiple published keys (specific values, regex, blank, present, not present, etc) as part of the subscription model. I could not figure this out using the routing_key and/or topic kind of stuff in RabbitMQ.
Check out Hutch (https://github.com/gocardless/hutch) - a similar system built on top of RabbitMQ
That is close. Thanks for the link.
The Consumer mixin is almost the same as the Subscriber one that I came up with.
Is there a way to subscribe on more than just the topic type? For example, to anything published that has { action: 'received' }
That's something that really enables some great use cases for us.
The Consumer mixin is almost the same as the Subscriber one that I came up with.
Is there a way to subscribe on more than just the topic type? For example, to anything published that has { action: 'received' }
That's something that really enables some great use cases for us.
Good example of resque/redis ecosystem usage. It could be great to have more info about your failover and monitoring strategies.
[deleted]
Currently all redis-based background job implementations does not support scheduled jobs.
Redis is not only a memcache replacement, but also a good data structure server, as central hub I hope it add more time-sturcture functions.