Ask HN: What do you use for job scheduling?

5 pointsby remotecode11 anni fa2 comments

Hi all,

I’m looking for a good job scheduler solution that fits my use case. In the past I have used the following:

  - Cron
  - SOS JobScheduler (http://www.sos-berlin.com/modules/cjaycontent/index.php?id=osource_scheduler_introduction_en.htm)
  - Celery with Celery Beat

All of these have pros and cons. None of these have satisfied the requirements that I have below, so I find myself building workarounds. SOS JobScheduler is the closest to meeting my requirements.

I’m curious what the standard tools for job scheduling are that people are using in production?

Here are my requirements:

  - Start a job at a given time
  - Start a job at a given interval (e.g. every 15 minutes)
  - Start a job on event:
    - Successful completion of prior job (e.g. a dependency)
    - Successful completion of multiple prior jobs
    - Failure of prior job
    - Keep all stderr and stdout for review
    - Track job run time (e.g. are my jobs getting slower?)
    - Alert on failure
    - Retry job

I would love to hear from the HN community what technologies you are using and how you like them / pros and cons.

2 comments

rajacombinator11 anni fa

Which ones of those use cases does Celery not fulfill? I currently use cron, which is pretty bad, and have an eye to move to Celery in the future. (Because it seems legit.) Will be pretty disappointed if it can't handle those reqs as they seem pretty standard for a job scheduler.

dilithiumhe311 anni fa

I would recommend using/looking at Amazon SQS. It seems to all (or atleast most) of your use cases and is managed so you won't need to deal with yet another system. It is strictly a message queue so you would still need to do some work to get things wired up but it should be straight forward.

Another mention, Amazon Simple Workflow is also something you may want to look at. I haven't used it in any project but seems like it could, sort-of work here. (not particularly sure though).