BashBooster – simple provisioning, bash only, requires nothing(bashbooster.net)
bashbooster.net
BashBooster – simple provisioning, bash only, requires nothing
http://www.bashbooster.net/
17 comments
> how hard could it be to write bash routines to download files, unpack them, install packages, etc.
I haven't used Gentoo for a long time but wasn't part of it's package system a nice little DSL for writing scripts that would "download files, unpack them, install packages, etc".
I haven't used Gentoo for a long time but wasn't part of it's package system a nice little DSL for writing scripts that would "download files, unpack them, install packages, etc".
Yep, on a full Gentoo environment one can make their own "ebuilds" which do all kinds of awesome things - I've got packages for web-app deploys and other fun things. Its python+bash
Ditto. And to the point BashBooster 'requires nothing'—it still requires bash, so good luck with cross-platform provisioning :)
On a more serious note, you should know how to use bash scripts effectively, but also know their limitations (true idempotence is hard, testing is annoying, error handling is hit or miss, etc.); there's a reason most infrastructure moves towards a higher level of abstraction/convenience like Ansible or Puppet.
On a more serious note, you should know how to use bash scripts effectively, but also know their limitations (true idempotence is hard, testing is annoying, error handling is hit or miss, etc.); there's a reason most infrastructure moves towards a higher level of abstraction/convenience like Ansible or Puppet.
A similar project designed for complex docker builds designed by me :)
http://ianmiell.github.io/shutit/
https://github.com/ianmiell/shutit
Used extensively within our corp for container builds, but easily repurposed for straightforward shell provisioning. Bash within a build-friendly python framework.
I talk about it more here:
https://www.youtube.com/watch?v=zVUPmmUU3yY
tho slightly out of date.
Some blogging on it here:
http://zwischenzugs.wordpress.com/
http://ianmiell.github.io/shutit/
https://github.com/ianmiell/shutit
Used extensively within our corp for container builds, but easily repurposed for straightforward shell provisioning. Bash within a build-friendly python framework.
I talk about it more here:
https://www.youtube.com/watch?v=zVUPmmUU3yY
tho slightly out of date.
Some blogging on it here:
http://zwischenzugs.wordpress.com/
This reminds me of fucking_shell_scripts[1] but this looks a lot simpler and less angry. I love the idea of truly having no dependencies, since I thought f_s_s was actually written in Bash but turns out to be written in Ruby. BashBooster on the other hand just relies on Bash (no gems, no Ruby) and looks easy to set up. I will definitely be trying it out.
[1]: https://github.com/brandonhilkert/fucking_shell_scripts
EDIT: BashBooster says it's developed to be used with Vagrant, whereas f_s_s is stand-alone. So it doesn't quite require nothing.
[1]: https://github.com/brandonhilkert/fucking_shell_scripts
EDIT: BashBooster says it's developed to be used with Vagrant, whereas f_s_s is stand-alone. So it doesn't quite require nothing.
Just noticed that it's using BitBucket to host the source code. Really wish the author chose GitHub instead, since network effect is a real thing and this will definitely make it harder for people like me to contribute back, not having a BB account nor willing to make one just to support a one-off project. Plus I just feel more comfortable navigating GitHub projects.
This comment may come off as overly angry but.. why the hell?!
In the time it took you to type your comment, you could have had a BB account registered, and in the time it takes to reply to my comment, you could have installed your SSH keys and had the same basic workflow set up.
People's unwillingness to explore alternatives for the most superficial of reasons are how monopolies are started and perpetuated.
Some day, Github is going to be acquired or taken over by advertisers or something, life is going to start sucking, and people will put up with it for the same reason they put up with Facebook.
In the time it took you to type your comment, you could have had a BB account registered, and in the time it takes to reply to my comment, you could have installed your SSH keys and had the same basic workflow set up.
People's unwillingness to explore alternatives for the most superficial of reasons are how monopolies are started and perpetuated.
Some day, Github is going to be acquired or taken over by advertisers or something, life is going to start sucking, and people will put up with it for the same reason they put up with Facebook.
The point was that network effect is real. And it's has nothing to do with the difficulty of opening an account. It's about discovery of open source projects. Try this: enter bitbucket.org, where's the search?
We're all on the same network, it's called the Internet. That's why we're talking about this project here.
That's a very peculiar way to do unit testing, comparing the output of the whole test instead of specific asserts.
How would you compare this with something like Shunit2[1]?
I can see someone missing some error confirmations by being careless when producing the output or testing things in isolation.
But I can also see how this could be more likely to catch some side effects (stupid example: unclosed terminal escapes) in combined output.
I was looking for a nice testing tool for my bash migration tool[2], and while the `assert` way is well known, I like writing bash because it's bare and requires discipline and yours seems survivalist enough.
[1]: https://code.google.com/p/shunit2/ [2]: https://github.com/zenoamaro/rash
How would you compare this with something like Shunit2[1]?
I can see someone missing some error confirmations by being careless when producing the output or testing things in isolation.
But I can also see how this could be more likely to catch some side effects (stupid example: unclosed terminal escapes) in combined output.
I was looking for a nice testing tool for my bash migration tool[2], and while the `assert` way is well known, I like writing bash because it's bare and requires discipline and yours seems survivalist enough.
[1]: https://code.google.com/p/shunit2/ [2]: https://github.com/zenoamaro/rash
I've seen it on many places to test shell scripts.
For example: in the same bash interpreter source code. See files using the '.right' extenion [0].
I think it's a flexible approach, compared to using things as shunit, or TAP producers.
The main advantage of using shunit (or TAP producers too), I think is to re-use available tools (at least in the case of TAP producers), and the possible familiarity of new contributors with the syntax.
Personally I avoid test libraries that use traps and eval, and bash code can become so complex, that I thing there is not a single testing library (Written in bash) that could work with all my code base.
As soon as you start using advanced features (shell options, traps, code tracing features, readonly stuff, filedescriptors, etc) most assumptions become wrong.
Anyway, the most hard part of test complex shell programs (specially those related to system administration), is to test functionality.
It's easy (and even not useful) to assert that 'myecho 1' will print '1' and return 0.
But things become hard to test when you want to see if a partition routine works well with +2TB disks, if a netfilter ruleset is resilient to attacks (unknown by the author), if a program is safe to unexpected (by the programmer) input, if a backup restore will not break production...
Fun to see this on hackernews front page. One of my pet projects is just the same, but not being inspired by chef.
[0] http://git.savannah.gnu.org/cgit/bash.git/tree/tests
For example: in the same bash interpreter source code. See files using the '.right' extenion [0].
I think it's a flexible approach, compared to using things as shunit, or TAP producers.
The main advantage of using shunit (or TAP producers too), I think is to re-use available tools (at least in the case of TAP producers), and the possible familiarity of new contributors with the syntax.
Personally I avoid test libraries that use traps and eval, and bash code can become so complex, that I thing there is not a single testing library (Written in bash) that could work with all my code base.
As soon as you start using advanced features (shell options, traps, code tracing features, readonly stuff, filedescriptors, etc) most assumptions become wrong.
Anyway, the most hard part of test complex shell programs (specially those related to system administration), is to test functionality.
It's easy (and even not useful) to assert that 'myecho 1' will print '1' and return 0.
But things become hard to test when you want to see if a partition routine works well with +2TB disks, if a netfilter ruleset is resilient to attacks (unknown by the author), if a program is safe to unexpected (by the programmer) input, if a backup restore will not break production...
Fun to see this on hackernews front page. One of my pet projects is just the same, but not being inspired by chef.
[0] http://git.savannah.gnu.org/cgit/bash.git/tree/tests
It's funny you mentioned the exact problems I'm trying to solve.
For the tests, I'm writing my own xUnit tool [1]. It doesn't use any fancy stuff, runs on set -euf, it's tested by itself and works on every POSIX-like shell that I could find (bash, dash, zsh, (m|pd)ksh, yash, posh, even busybox) and many different versions and OSs.
On the functional side, I've decided that I need to test using virtual machines. The test matrix tool that can spawn and test on them is almost ready [2], it already tests the library itself and exports the matrix configuration to Travis.
Very early stuff though and doesn't even touch problems like partition routines you mentioned, but overall I'm happy with the results, considering everything runs under 70Kb of code.
[1] https://github.com/Mosai/workshop/blob/master/doc/posit.md
[2] https://github.com/Mosai/workshop/blob/master/test/matrix.sh
For the tests, I'm writing my own xUnit tool [1]. It doesn't use any fancy stuff, runs on set -euf, it's tested by itself and works on every POSIX-like shell that I could find (bash, dash, zsh, (m|pd)ksh, yash, posh, even busybox) and many different versions and OSs.
On the functional side, I've decided that I need to test using virtual machines. The test matrix tool that can spawn and test on them is almost ready [2], it already tests the library itself and exports the matrix configuration to Travis.
Very early stuff though and doesn't even touch problems like partition routines you mentioned, but overall I'm happy with the results, considering everything runs under 70Kb of code.
[1] https://github.com/Mosai/workshop/blob/master/doc/posit.md
[2] https://github.com/Mosai/workshop/blob/master/test/matrix.sh
I'm not really in charge of managing companies infrastructure, so it doesn't stand so important for me, but every time I run across talking about all these provisioning tools, I'm taken aback by how many of them are there. Ansible, Puppet, Chef, now this BashBooster. All of them written in different languages, having different syntax and all doing basically the same thing — the same thing I'm using a couple of bash-scripts for when I need to set up another server or VM!
Yet all projects are alive yet, so there probably is some difference which makes all tools applicable. So it makes me feel uncomfortable asking this obviously nonsensical question, but: is there some one tool I should chose if I want to get away from the stone age of using bash scripts for provisioning?
Yet all projects are alive yet, so there probably is some difference which makes all tools applicable. So it makes me feel uncomfortable asking this obviously nonsensical question, but: is there some one tool I should chose if I want to get away from the stone age of using bash scripts for provisioning?
Depends what you're trying to do. If you're a programmer and your use case is simple, then a few bash scripts are fine.
Ansible is a great alternative to Chef/Puppet etc. Agentless and runs over ssh.
Depends on your usecase / scale of deployment. Chef/Puppet being pull oriented and having a central server to query for data are often usable where Ansible is not an good alternative. (the ssh and agentless part make it a worse option)
For simple standing up of a new box though - sure, any of them will do.
For simple standing up of a new box though - sure, any of them will do.
You can set up Ansible to do pull if you really want to and you can there are other solutions for doing dynamic inventory, gathering facts, or storing data on other services.
You can set up Ansible to automatically refresh repository and run scripts locally. That's only a small bit of what chef/puppet offer. And sure, there are add-ons to cover other bits one by one. But that's different from the system offering centralised inventory and automated pull of code/data on its own - otherwise you could just as well say "you can set up bash to do pull if you really want to and there are other solutions for ...". But it's just not the same.
Yet it's a Mercurial repo. Not a big deal and I have nothing against Mercurial, but something that's marketed with "no dependencies" would be better off if it was on Git - way more accessible.
Actually, you can just download zipped project directory on bitbucket. And it sure won't have less dependencies if "dependency" will be git instead of mercurial.
The number of attempts to fix provisioning is overwhelming.
Shows that chef/puppet aren't cutting it, and neither are the smaller alternatives.
We need to use it in combination with phoenix deployments. Moving target builds are doomed to failure:
https://zwischenzugs.wordpress.com/2014/06/07/phoenix-deploy...
https://zwischenzugs.wordpress.com/2014/06/07/phoenix-deploy...
I build a similar library that does exactly this and have used it internally for a couple of years. The approach complements ansible quite nicely. For simple setups it is easier to read and modify. It is also less learning curve for the other developers.
I recently started using bork[1] which is a bash-based DSL with a similar philosophy. Folks interested in BashBooster should also check it out.
[1]: https://github.com/mattly/bork
[1]: https://github.com/mattly/bork
Same sort of thing that I've been deploying everything with. http://github.com/AndrewVos/garrison
Something similar from visionmedia (written in go): https://github.com/visionmedia/stack
This isn't very abstracted from running the commands directly. For example the package management differs for yum and apt. I do appreciate simplicity though.
Reminds me of my POSIX shell configuration management thingy: http://git.uggedal.com/conf
I've thought about doing similar things in the past (I like the idea of deploying with bash/extended-bash) - great to see this!
I looked at the code for bb-sync(). I just used cp and touch. Would be cool if it checked for rsync and used it.
Nice toolkit, will be using, thanks!
After 3 weeks or labouring and using FSS with a fully working system, I dropped it all and went back to Ansible. I don't regret the time spent because I picked up a lot of transferrable skills from command line utilities and bash.
There are simple things like downloading a file which BashBooster gets wrong. If the file already exists, it should not abort, it should resume download or compare file sizes to ensure that the previous download didn't terminate abruptly. Basically, this and other idempotency issues need to be dealt with very carefully. Even simple things like "mkdir -p foo" instead of just "mkdir foo".
Once you get past idempotency you will most likely run into performance issues and then into project velocity.
If you really must use Bash Scripts, I would suggest using them via Ansible's Shell + Command modules. This will allow you to lean on Ansible until your Bash scripts can catch up. It will also give you Ansible's inventory management features.