Shovel: Rake for Python(github.com)
github.com
Shovel: Rake for Python
https://github.com/seomoz/shovel
11 comments
As a user of and sometimes contributor to fabric, I would like to point out that fabric is really good, but not perfect. No software really is. There are times I have chafed against fabric -- sometimes I would like it to be "heavier" doing more things automatically, and there are others I would like it to be "lighter", getting out of my way so I can just do my stuff. I will still reach for fabric first when I need tools, but it is nice to have alternatives available for those times I need something else.
Similarly, it frequently benefits all projects to have a few alternatives and different implementations for the same concept. Ideas can be explored independently and features can be borrowed back and forth. If nothing else, and provided that shovel gets to a fuller development stage, this could be a good thing for all.
Similarly, it frequently benefits all projects to have a few alternatives and different implementations for the same concept. Ideas can be explored independently and features can be borrowed back and forth. If nothing else, and provided that shovel gets to a fuller development stage, this could be a good thing for all.
The only thing I don't like about Fabric ( and perhaps I missed reading a step in the docs ) is working with very complex systems, I wanted to break fabric recipes/scripts into multiple files and found doing `fab -f zkservers.py reload` slightly tedious. I've much rather do `./zkservers.py reload`
> Being able to have non-techies run Fabric tasks is definitely a goal that's passed me fleetingly in the night.
Isn't that equipping monkeys with guns? We all know what happens when you do that: http://www.youtube.com/watch?v=AabGdnTxqD0
Isn't that equipping monkeys with guns? We all know what happens when you do that: http://www.youtube.com/watch?v=AabGdnTxqD0
What does this solve that fabric doesn't? Read through the docs but didn't see any comparisons.
It doesn't seem to do task dependency like Rake, which is unfortunate. It's the only thing missing from fabric.
I'm half a mind to write my own with pbs (https://github.com/amoffat/pbs) thrown in...
I'm half a mind to write my own with pbs (https://github.com/amoffat/pbs) thrown in...
See absolutely no overwhelming cause to invest time switching from Fabric to this. I think everyone (including fabric maintainers) would love some alternative solutions in this space but this seems like a lateral move.
We already have Paver[1] in this space. It has task dependencies, can replace distutils, is used at Mozilla.
[1] http://paver.github.com/paver/
[1] http://paver.github.com/paver/
One thing Shovel has going for it is that it appears to "just work" for Windows users. Compare to the setup process for Fabric:
http://www.jonnyreeves.co.uk/2011/08/getting-python-and-fabr...
Windows doesn't support hashbangs, so Shovel does currently need a batch file in Python's Scripts directory, but that's not a huge deal.
http://www.jonnyreeves.co.uk/2011/08/getting-python-and-fabr...
Windows doesn't support hashbangs, so Shovel does currently need a batch file in Python's Scripts directory, but that's not a huge deal.
WHY CAN'T WE JUST USE MAKEFILES
Cause sometimes you need your tasks to have, you know, logic and stuff ? Or am I the only one who want to do the equivalent of a "if" in a build script from times to times, and feel completely helpless with Make ?
Thanks. Did not knew about this (I should have pointed that I haven't been using Makefiles for a while).
However :
"Conditionals control what make actually "sees" in the makefile, so they cannot be used to control shell commands at the time of execution." (emphasis mine).
So I get that is would not solve the kind of issues I had in mind... Never mind.
However :
"Conditionals control what make actually "sees" in the makefile, so they cannot be used to control shell commands at the time of execution." (emphasis mine).
So I get that is would not solve the kind of issues I had in mind... Never mind.
You're not supposed to put significant logic in a makefile. You've already got tons of tools for real logic. A makefile is a dependency graph. If statements are for defining slightly different graphs in slightly different environments.
If you need to add real logic, write a shell script.
A well written Makefile has a small preamble of variable definitions, followed by a couple of named convenience .PHONY targets (without any commands), followed by a small handful of pattern rules, followed by the dependency graph. Each edge in the dependency graph should include as few commands as possible, preferably just one!
If you need to add real logic, write a shell script.
A well written Makefile has a small preamble of variable definitions, followed by a couple of named convenience .PHONY targets (without any commands), followed by a small handful of pattern rules, followed by the dependency graph. Each edge in the dependency graph should include as few commands as possible, preferably just one!
While perhaps not eloquently stated, the point is valid. Makefiles work as well as anything for most of these sorts of tasks.
> Makefiles work as well as anything
I'd argue that they always work better than any of these language-specific alternatives ever do.
I'm not saying that GNUMake or BSD's Make or any particular Make is perfect. Surely there is still a lot of evolution to happen in build systems.
What I am saying is that if you have a bunch of scripts to run & a bunch of files they produce, you can't beat a general purpose dependency graph with files for nodes and command lines for edges.
Lately, I `touch Makefile` as one of the absolute first things I do on a project, regardless of language or framework, even when just hacking around. I'll even write a Makefile if I'm just doing some data analysis, like multi-stage log grepping. I find `rm intermediate.file; make` to be a nice alternative to putzing with command history and avoiding waisting time redoing expensive steps.
Hell, we completely manage the configuration on our servers using Make instead of something like Puppet or Chef. So far, it's working pretty well, bit I'm still a newbie at sysadmin stuff.
We've even replaced all of the db migration & other common Rails rake tasks with a few short bash scripts and ~10 lines of Make. We use .sql files for migrations and store db/schema.sql instead of db/schema.rb. It's two orders of magnitude faster to run our tiny script/dump-schemas.sh against psql instead of loading the Rails environment to query the db.
I'd argue that they always work better than any of these language-specific alternatives ever do.
I'm not saying that GNUMake or BSD's Make or any particular Make is perfect. Surely there is still a lot of evolution to happen in build systems.
What I am saying is that if you have a bunch of scripts to run & a bunch of files they produce, you can't beat a general purpose dependency graph with files for nodes and command lines for edges.
Lately, I `touch Makefile` as one of the absolute first things I do on a project, regardless of language or framework, even when just hacking around. I'll even write a Makefile if I'm just doing some data analysis, like multi-stage log grepping. I find `rm intermediate.file; make` to be a nice alternative to putzing with command history and avoiding waisting time redoing expensive steps.
Hell, we completely manage the configuration on our servers using Make instead of something like Puppet or Chef. So far, it's working pretty well, bit I'm still a newbie at sysadmin stuff.
We've even replaced all of the db migration & other common Rails rake tasks with a few short bash scripts and ~10 lines of Make. We use .sql files for migrations and store db/schema.sql instead of db/schema.rb. It's two orders of magnitude faster to run our tiny script/dump-schemas.sh against psql instead of loading the Rails environment to query the db.
Might as well asks "why use Fabric/Capistrano/Chef/etc" instead of Makefiles. We have progressed beyond that, and beyond caps-lock only keyboards.
Why not "just use makefiles".
Because they use a BS syntax and are not as convenient as Python.
Because they are not even portable, whereas Python scripts run on Windows just fine.
Because they weren't even invented for that purpose, but for compiling stuff, and they failed even that so much, that every other project has it's own implementation (cmake, qmake, ant, scons, ...) or at the very least uses automation tools on top make, like the autotools, etc.
Why not "just use makefiles".
Because they use a BS syntax and are not as convenient as Python.
Because they are not even portable, whereas Python scripts run on Windows just fine.
Because they weren't even invented for that purpose, but for compiling stuff, and they failed even that so much, that every other project has it's own implementation (cmake, qmake, ant, scons, ...) or at the very least uses automation tools on top make, like the autotools, etc.
N downvotes from people suggesting we use Makefiles for the purposes we use Rake etc? And they call themselves "hackers"?
Probably the embodiment of all that's wrong with legacy cruft and voodoo practices.
Probably the embodiment of all that's wrong with legacy cruft and voodoo practices.
No, it's mostly because the Makefile grammar is incredibly well suited to any of the things you've mentioned -- it's just seen as "old", and most people don't know it anymore.
It's neither legacy nor voodoo. Your statement that if you suggest using one tool in place of another makes you something besides a hacker is incomprehensible.
PUppet/chef/capistrano are specialized versions of Make. Rake is a misnomer, as it's just a collect of "tasks". Again, it's a subset of what Makefiles can do.
It's a little bit like using Lisp -- it's a "dilapidated", "old" tool but most modern tools are subsets of it in some manner.
It's neither legacy nor voodoo. Your statement that if you suggest using one tool in place of another makes you something besides a hacker is incomprehensible.
PUppet/chef/capistrano are specialized versions of Make. Rake is a misnomer, as it's just a collect of "tasks". Again, it's a subset of what Makefiles can do.
It's a little bit like using Lisp -- it's a "dilapidated", "old" tool but most modern tools are subsets of it in some manner.
No, it's mostly because the Makefile grammar is incredibly well suited to any of the things you've mentioned -- it's just seen as "old", and most people don't know it anymore.
It's not just seen as old. It is old, and not in a good way. The only thing it has got going for it, is being generally available.
Make has poor support for many common needs. For example, it doesn't track dependencies itself. It lacks proper namespace support. The whitespace rules can get you. It lacks higher level constructs that one can build with other tools. You can't programmatically introspect the make script to see how it goes or debug it if it behaves unexpectedly. It doesn't know anything about the stuff you're working with -- i.e you're stuck with plain old command line instead of smart context based scripting ("now I want to update some configuration file", "now I want to sftp stuff", "now I want to ... on a non POSIX machine", "now I want to interface with a database"). It's also bad for managing multiple different environments, leading to the autotools et co development.
Your statement that if you suggest using one tool in place of another makes you something besides a hacker is incomprehensible.
A hacker should use the right tool for the right job. So, yes, using one tool in place of another (that fits better) makes you less of a hacker, at least in the tool selection aspect.
PUppet/chef/capistrano are specialized versions of Make. Rake is a misnomer, as it's just a collect of "tasks". Again, it's a subset of what Makefiles can do.
And C, Lisp, Haskell are just specialized versions of what assembly can do. This is beside the point, which is those tools are better for their perspective uses (and the tool mentioned in the post link is not about building C projects).
It's not just seen as old. It is old, and not in a good way. The only thing it has got going for it, is being generally available.
Make has poor support for many common needs. For example, it doesn't track dependencies itself. It lacks proper namespace support. The whitespace rules can get you. It lacks higher level constructs that one can build with other tools. You can't programmatically introspect the make script to see how it goes or debug it if it behaves unexpectedly. It doesn't know anything about the stuff you're working with -- i.e you're stuck with plain old command line instead of smart context based scripting ("now I want to update some configuration file", "now I want to sftp stuff", "now I want to ... on a non POSIX machine", "now I want to interface with a database"). It's also bad for managing multiple different environments, leading to the autotools et co development.
Your statement that if you suggest using one tool in place of another makes you something besides a hacker is incomprehensible.
A hacker should use the right tool for the right job. So, yes, using one tool in place of another (that fits better) makes you less of a hacker, at least in the tool selection aspect.
PUppet/chef/capistrano are specialized versions of Make. Rake is a misnomer, as it's just a collect of "tasks". Again, it's a subset of what Makefiles can do.
And C, Lisp, Haskell are just specialized versions of what assembly can do. This is beside the point, which is those tools are better for their perspective uses (and the tool mentioned in the post link is not about building C projects).
Interesting, but at this stage the only functionality it provides is tasks discovery and argument passing.
What I would like to see is some facility to aid with writing meaningful tasks that execute system utilities or perform work on remote servers over ssh.
The Web API is also a nice concept, but I would prefer a control panel over Fabric-like tool rather than a dependency that I have to install on every target host.
What I would like to see is some facility to aid with writing meaningful tasks that execute system utilities or perform work on remote servers over ssh.
The Web API is also a nice concept, but I would prefer a control panel over Fabric-like tool rather than a dependency that I have to install on every target host.
Right now it seems more comparable to argparse than Rake or fabric. The server part is neat, but a lot of my fabric tasks would probably time out in the browser (unless this can run tasks in the background, async).
it seems the whole point is simplicity
> Tasks should be easy to define -- one decorator, no options
lets see how long it lasts... at some point they will realize they need a build-tool. Rake is a build-tool. Shovel is not.
> Tasks should be easy to define -- one decorator, no options
lets see how long it lasts... at some point they will realize they need a build-tool. Rake is a build-tool. Shovel is not.
Python already has tools for this. IF I were interested in my working environment becoming more like a Ruby working environment, why wouldn't I just use Ruby?
Having said that, it looks like the HTTP server is the USP here. Being able to have non-techies run Fabric tasks is definitely a goal that's passed me fleetingly in the night. So, add this to Fabric and we're good.
EDIT: The author, Dan Lecocq is near me in Seattle so, what with it being a small world and all, I just wrote to him (in case he's not the author of this thread) via his blog to ask him what he didn't like about Fabric, and suggested he respond here.