Sidekiq Pro: advanced functionalities for Sidekiq(mperham.github.com)
mperham.github.com
Sidekiq Pro: advanced functionalities for Sidekiq
http://mperham.github.com/sidekiq-pro/
3 comments
I do wish Sidekiq had an option for using postgresql for the job store (like queue_classic: https://github.com/ryandotsmith/queue_classic)
It would be a decent feature, but I understand mperhams argument (https://github.com/mperham/sidekiq/wiki/FAQ). And Redis is lightweight and reliable enough that adding it onto any project is fairly painless.
Not to mention when you're already using Redis for sidekiq you tend to find some other solutions work really well built on top of it.
Not to mention when you're already using Redis for sidekiq you tend to find some other solutions work really well built on top of it.
I've been using Sidekiq for a while now and also purchased the pro license. But using redis as a store is only nice when everything is going well. I've managed to have production blow up on me because cascading job failures caused redis to exceed memory. RAM is pretty tight on EC2 instances. And when redis runs out of memory, it restarts and reloads the DB, meaning I'm going to end up failing catastrophically again soon after.
I've worked around that now, applying some similar workarounds I had kicking around for resque. But it'd be nice to use a different store where I can fall back to disk if needed. The jobs are already async, so I've accepted the nominal overhead of disk I/O.
I've worked around that now, applying some similar workarounds I had kicking around for resque. But it'd be nice to use a different store where I can fall back to disk if needed. The jobs are already async, so I've accepted the nominal overhead of disk I/O.
I second the wish to see PG supported here!
I wanted to use Redis for my SaaS product (I have been using it for behind-LAN apps for a long time) but realized that in 3 setups I studied (RedisToGo, DotCloud and a third I can't remember), you are (if I'm right, and by default) basically exposing the Redis instance to the internet directly, which is not advised at all [1].
So having the PG option here would be fairly interesting.
Please note that the above is just my current comprehension after some research in limited time - I'd love to be wrong here.
[1] http://redis.io/topics/security
I wanted to use Redis for my SaaS product (I have been using it for behind-LAN apps for a long time) but realized that in 3 setups I studied (RedisToGo, DotCloud and a third I can't remember), you are (if I'm right, and by default) basically exposing the Redis instance to the internet directly, which is not advised at all [1].
So having the PG option here would be fairly interesting.
Please note that the above is just my current comprehension after some research in limited time - I'd love to be wrong here.
[1] http://redis.io/topics/security
You can setup http authentication to your redis instance which provides at least some sort of security. Also, you can lock your servers down to only accept connections from specific IPs (so like your app servers are the only ones allowed access to your Redis server). Those two things combined are a good start for security at the very least.
The culprit here is that restricting connections is often not possible at all if the Redis provider is different from the provider used for the rest of the app (eg: using Heroku/DotCloud + RedisToGo or similar, etc) - that's my main issue here.
You can do this if you use AWS security groups, or if your PaaS support similar stuff, but not all do support it so far, if I read right...
You can do this if you use AWS security groups, or if your PaaS support similar stuff, but not all do support it so far, if I read right...
Redis is really really easy to host yourself. I'm not much of a fan of outsourcing infrastructure items though, so I am biased
I'm not contesting that Redis is easy to host yourself - I host it myself on some projects, use Chef to provision servers etc.
If you choose to use the cloud though (PaaS etc) to save time on the rest of your app, then pick a manually managed server for Redis, it kind of defeat the purpose, plus you won't be able to easily follow the changes on the rest of the PaaS infrastructure (eg: change of servers with access right etc).
If you choose to use the cloud though (PaaS etc) to save time on the rest of your app, then pick a manually managed server for Redis, it kind of defeat the purpose, plus you won't be able to easily follow the changes on the rest of the PaaS infrastructure (eg: change of servers with access right etc).
Yes. But it's much simpler to rely the existing security protections you already use for your database, instead of having to setup new ones with a completely different system for your async jobs.
There's a few benefits to using postgresql for these jobs:
1. Atomic backups. You can backup your jobs at the same time you backup your data. If you need to restore the data and jobs, you get a consistent snapshot.
2. Less moving parts. If all you need is postgresql, that simplifies things tremendously.
3. Transactional job control. You can add jobs inside a transaction and roll them back if needed. You can't do that easily with redis. With activerecord, you have to use after_commit, which has some complications and potential for dataloss.
1. Atomic backups. You can backup your jobs at the same time you backup your data. If you need to restore the data and jobs, you get a consistent snapshot.
2. Less moving parts. If all you need is postgresql, that simplifies things tremendously.
3. Transactional job control. You can add jobs inside a transaction and roll them back if needed. You can't do that easily with redis. With activerecord, you have to use after_commit, which has some complications and potential for dataloss.
Great news and great piece of software.
By far the best async job framework available for Ruby imho.