Schedule – Python job scheduling for humans(github.com)
github.com
Schedule – Python job scheduling for humans
https://github.com/dbader/schedule
11 comments
I've been talking with a few developer friends about libraries lately, and this one has made me realize what they're saying. As Python developers, DSLs are completely not in our DNA, but this library strikes me as something that would look much more Pythonic with a DSL.
Wouldn't it be better to have descriptions of:
Every day at 10:30 do myfunc
Every Monday do otherfunc
Every hour do whatever
Rather than the current .every().day.at("10:30").do(myfunc)?
Wouldn't it be better to have descriptions of:
Every day at 10:30 do myfunc
Every Monday do otherfunc
Every hour do whatever
Rather than the current .every().day.at("10:30").do(myfunc)?
I find it hard to appreciate these inventive types of API design. Something more traditional and boring, like Celery's crontab[1] or dateutils rrule[2] seem quite sufficient. For example:
[2]: https://labix.org/python-dateutil#head-470fa22b2db72000d7abe...
crontab(hour=18, minute=30, day_of_week='*')
[2]: https://labix.org/python-dateutil#head-470fa22b2db72000d7abe...
I'm not sure why, given two equally flexible designs, we should choose the least expressive. The fact that cron is sufficient doesn't mean I don't still have to google how to run things every three hours every second Monday, for example.
I agree. This scheduler dialect for Rebol is pretty much the DSL you describe - http://softinnov.org/rebol/scheduler.shtml
I'd rather have neither. I particularly dislike the .every().day.at("10:30").do(myfunc) thing where methods and attributes are abused in order to artificially make it read like a sentence.
This is not Pythonic at all <troll>it's like some childish Ruby code</troll>
This is not Pythonic at all <troll>it's like some childish Ruby code</troll>
We prefer to use Celery to schedule periodic tasks [1] and are moving most of our cron jobs to Celery because it is easier to monitor, test, retry, and send reports on failed jobs.
We extended [2] django-celery db backend to store enough information about the periodic tasks to retry them directly in Django admin or to filter the results (we support storing the task results in JSON instead of pickle).
Celery is probably more difficult to configure but is far more flexible and robust than this lib...
[1] http://celery.readthedocs.org/en/latest/userguide/periodic-t... [2] https://github.com/resulto-admin/django-celery-fulldbresult
We extended [2] django-celery db backend to store enough information about the periodic tasks to retry them directly in Django admin or to filter the results (we support storing the task results in JSON instead of pickle).
Celery is probably more difficult to configure but is far more flexible and robust than this lib...
[1] http://celery.readthedocs.org/en/latest/userguide/periodic-t... [2] https://github.com/resulto-admin/django-celery-fulldbresult
My team likes Celery to do stuff like this always which is indeed better/necessary for tasks which need to be executed always even when web server dies/restarts. For a lot of jobs that is not needed (where the background is directly related to the foreground and without the foreground process working properly the background processes make no sense anymore) however and then Celery is a pain in the ... to setup & maintain properly and, frankly, overkill, even with the simple setup solutions like Redis. I would recommend in process schedulers for those cases as they require no setup.
So - this is interesting. The thing that I always look for is how the system deals with all of the time issues that come up. For example, how does it deal with Daylight Savings, or a leap second, or adjustments to time, or the daemon getting restarted, etc. I don't see anything specific to this library that handles these scenarios.
Is there any system out there that has a good design for handling these issues?
Is there any system out there that has a good design for handling these issues?
My issue is how to make this distributed (unless Inmissed that)
Have many out of process (on disk) files that contain some or all of the phrases "start package a with Params b at time c"
And have the scheduler pass over the actual running to another process - possibly another machine entirely
Would like to get my teeth into that
Have many out of process (on disk) files that contain some or all of the phrases "start package a with Params b at time c"
And have the scheduler pass over the actual running to another process - possibly another machine entirely
Would like to get my teeth into that
I like the feel of it and the thought behind it, but ... while true, sleep?
I get very uncomfortable with any kind of polling, especially polling timers for checking for jobs with timespans between them. Maybe it's just me?
I get very uncomfortable with any kind of polling, especially polling timers for checking for jobs with timespans between them. Maybe it's just me?
every cron or cron-like service has some kind of polling behind. Beanstalk, redis-queue, you name it. Many open source redis based scheduler are all implemented in a sleep(5) loop.
Even the Unix cron is implemented with a sleep() call
https://en.wikipedia.org/wiki/Cron#History
Even the Unix cron is implemented with a sleep() call
https://en.wikipedia.org/wiki/Cron#History
As another commentor noted, this is the way every system is implemented. It is simple, slightly inefficient, and deals with most of the nasty problems that arise when dealing with time on all systems today.
Currently I use APScheduler for this exact use-case. This looks interesting and I will be taking a closer look at it.
Can this be used in production?
You'll need to wrap it in `threading` to avoid block your main thread, or use greenlet, or wrap it as a RESTful service, then you have the problem of high-availability, so a distributed shared consensus system would be better, the next thing you know you have NTP issues... The problem goes on.
Job scheduling not really rocket science, so sure.
Job scheduling when things fail is harder
Job scheduling when things fail is harder
Why is the time.sleep(1) needed inside the while loop?
Aside: I'm not a fan of the "for humans" label that seems to be common among some Python projects now.
It implies the other tools are possibly designed by monkeys, to a degree, which is often totally not the case.
As a particular example, while some libs are "friendly", they are also limited.
In this case, celery is implied to be not "for humans", but it actually offers distributed job queuing and jobs that can survive a reboot, which is pretty darn important.
I'm not going to name names of other libs doing this, but there are a few that have picked API simplicity over complete feature set. This is obviously a balance - BUT I think it unfairly casts the alternatively libs in a questionable light, and often they can do a lot more.
FWIW, the Ruby-design into the method space can also be reduced if desired to make this a bit more Pythonic (though some things are personal preferences):
One thing I also notice (not sure if Ruby inspired or not), is that it's keeping global state in the module (see run_pending) rather than having different scheduler objects. This somewhat limits reusability by having this embedded magic and not being able to isolate schedulers.
It implies the other tools are possibly designed by monkeys, to a degree, which is often totally not the case.
As a particular example, while some libs are "friendly", they are also limited.
In this case, celery is implied to be not "for humans", but it actually offers distributed job queuing and jobs that can survive a reboot, which is pretty darn important.
I'm not going to name names of other libs doing this, but there are a few that have picked API simplicity over complete feature set. This is obviously a balance - BUT I think it unfairly casts the alternatively libs in a questionable light, and often they can do a lot more.
FWIW, the Ruby-design into the method space can also be reduced if desired to make this a bit more Pythonic (though some things are personal preferences):
scheduler = schedule.scheduler()
scheduler(callback=job).every(minutes=10)
scheduler(callback=job).every(day=1).at(hour=10, minutes=30)
One thing I also notice (not sure if Ruby inspired or not), is that it's keeping global state in the module (see run_pending) rather than having different scheduler objects. This somewhat limits reusability by having this embedded magic and not being able to isolate schedulers.
"For humans" is surely intended to be the opposite of "for computers". It's meant to say "When we designed this thing, we made it a high priority for it to be easy to learn, easy to remember, and easy to use".
There's certainly an implication that other libraries are designed without sufficient attention to human factors, and maybe that's kinda rude, but the same goes for making any positive claim. "A library for efficient high-precision computations": are all the others inefficient? "A flexible and powerful tool for retrieving structured data from web APIs": are all the others inflexible and weak? "A standard-conformant C++ parser": are all the others broken? (None of those examples, so far as I know, are from real projects, but clearly they all could be.)
There's certainly an implication that other libraries are designed without sufficient attention to human factors, and maybe that's kinda rude, but the same goes for making any positive claim. "A library for efficient high-precision computations": are all the others inefficient? "A flexible and powerful tool for retrieving structured data from web APIs": are all the others inflexible and weak? "A standard-conformant C++ parser": are all the others broken? (None of those examples, so far as I know, are from real projects, but clearly they all could be.)
I also find the "for Humans" thing annoying, but not for the reasons you list. The examples you list are all specific and informative. In the case of "for Humans" what is probably meant is something like "straightforward" or "simple". The thing is, there isn't anything that is NOT designed "for Humans", so the label is meaningless, IMHO.
Monkeys? Where did that come from? The idea of "for humans" just means that the UI was developed with an humane approach - as opposed to an undesigned UI, or one designed merely for the simplicity of the implementation, and that doesn't take into consideration things like our need for consistency to make up for a non-perfect memory.
That said, I personally don't think Celery suffers from this problem, but API that was being replaced by the first project to used this moniker - urllib2 - definitively did.
That said, I personally don't think Celery suffers from this problem, but API that was being replaced by the first project to used this moniker - urllib2 - definitively did.
It's a stand in for something "not for humans". On second thought, monkeys would require an even easier UI. Perhaps "people that like pain".
I don't think it's fair to say the other tools weren't designed, or merely designed the easy way.
urllib2 has some rough edges - and some things should be better, but it can also do some things requests can't do, and I've had to reach for it on a few occasions. requests has basically optimized for the easy case, where
urllib2 wished to be more flexible in certain areas, so it deserves some merit there too.
I don't think it's fair to say the other tools weren't designed, or merely designed the easy way.
urllib2 has some rough edges - and some things should be better, but it can also do some things requests can't do, and I've had to reach for it on a few occasions. requests has basically optimized for the easy case, where
urllib2 wished to be more flexible in certain areas, so it deserves some merit there too.
> API that was being replaced by the first project to used this moniker - urllib2 - definitively did.
For curious: that was python library requests. And the claim was more than deserved in that case.
For curious: that was python library requests. And the claim was more than deserved in that case.
> "for humans"
I think in recent years, Kenneth Reitz has really taken "for humans" and run with it: "Python for Humans", "Deployment for Humans", "requests: HTTP for Humans". He's even gone so far as to (facetiously?) trademark it ("args: Argument Parsing for Humans™"). Now everything with syntactic sugar is "for humans".
If you think your API represents a breakthrough in simplicity then trust me: it will do the talking for you.
Note to self: don't describe strider (https://github.com/mpdehaan/strider) as Vagrant for humans.
The global scheduler object is provided for convenience, it is possible to create others.
I agree about the for humans branding.
I also don't find this api very appealing, but I haven't given much consideration to what an interval api should look like.
I agree about the for humans branding.
I also don't find this api very appealing, but I haven't given much consideration to what an interval api should look like.
It should probably look a lot like ISO 8601.
In fact, it can be quite difficult to reason about. What, for example is "schedule. every(10)"? By itself, it does not seem to represent a useful concept other than a "partial" schedule statement.
I would prefer to use an api using keyword arguments to set the various (optional) parameters to build a schedule.