Shell programming with bash: by example, by counter-example(matt.might.net)
matt.might.net
Shell programming with bash: by example, by counter-example
http://matt.might.net/articles/bash-by-example/
10 comments
I was on the fence about adding [[ ]].
I think I'll add it in a revision.
I also agree on bash being quirky. I tell my class that the gold standard is a bash script that works 99% of the time.
I think I'll add it in a revision.
I also agree on bash being quirky. I tell my class that the gold standard is a bash script that works 99% of the time.
Seconded for [[ and ]]. Also { and } (with semi colons) vs ( and ), for sub processes would be nice to add.
I'd also note that your discussion of scope doesn't mention local variables are possible in bash, see: http://www.tldp.org/LDP/abs/html/localvar.html
I think it might be lacking content on idioms.
Things like 'while read -r line; do {stuff with $line}; done'
These idioms have issues for their most common uses (e.g. dealing with filenames with embedded newlines), but they do work 99% of the time.
Things like 'while read -r line; do {stuff with $line}; done'
These idioms have issues for their most common uses (e.g. dealing with filenames with embedded newlines), but they do work 99% of the time.
They work 100 % of the time if you do it right. An example:
Of course, in this case it would be better to use:
find . -type f -name foo -print0 | while read -rd ''; do
grep -l bar "${REPLY}"
done
Of course, in this case it would be better to use:
find . -type f -name foo -print0 | xargs -0 -I{} grep -l bar '{}'
http://mywiki.wooledge.org/BashFAQ/031 is a great reference.
If your guide is for bash and not sh, then I would say don't teach them [, just do [[. It's safer, since it won't glob or split words, you don't need quoting, and you don't need to escape operators like < and >. Maybe it's can be seen as more difficult since it's a sublanguage (doesn't use the regular syntax rules), but so are (( and $(( which you've discussed at length, and which are typically less useful than [[ in bash.
If your guide is for bash and not sh, then I would say don't teach them [, just do [[. It's safer, since it won't glob or split words, you don't need quoting, and you don't need to escape operators like < and >. Maybe it's can be seen as more difficult since it's a sublanguage (doesn't use the regular syntax rules), but so are (( and $(( which you've discussed at length, and which are typically less useful than [[ in bash.
What to use instead of bash though? I hate having to worry about all the little quirks in bash but most other languages make simple things like input redirection and piping commands awfully verbose.
Check out IPC::Run for Perl. [0] Perl was already pretty nice with it's backticks but IPC::Run takes it another step further and allows you to do all sorts of redirection/piping/etc with the single 'run' subroutine. If you want an 'expect' sort of use, you can do that too.
[0]: http://search.cpan.org/~toddr/IPC-Run-0.92/lib/IPC/Run.pm
[0]: http://search.cpan.org/~toddr/IPC-Run-0.92/lib/IPC/Run.pm
I use fish.
http://fishshell.com/
Loop:
http://fishshell.com/
Loop:
for f in *.txt
cat $f
endReally liking it so far. Thanks for the suggestion.
Haskell! Seriously, people do shell scripting in Haskell. There's actually a pretty nice library designed for it called Shelly:
https://github.com/yesodweb/Shelly.hs/blob/master/README.md#...
https://github.com/yesodweb/Shelly.hs/blob/master/README.md#...
Shelly is great! Still, there is some minimum complexity before something is better done in it, and porting a small bash script that has grown from a tiny bash script is pretty trivial.
rc is pretty minimal and consistent (available in plan9ports).
I try to keep my scripts extensions of what I'd type at the prompt - often quite directly, yanking it from my history when I realise I need to do something frequently or pulling a line into my editor with with edit-and-execute-command when it gets big (for a single line) and saving it off somewhere. Once it gets above about 10 lines, it's usually time for something else, though I've boiled a few frogs in my day...
Modern Bash (a.k.a. the bash you get with Linux but not with OS X) even has associative arrays similar to what you would call a dict in Python or a hash in Perl.
$ declare -A capitals
$ capitals["Illinois"]="Springfield"
$ capitals["California"]="Sacramento"
$ echo ${capitals["California"]}
Sacramento
$ echo ${capitals["Illinois"]}
Springfield
$ for k in ${!capitals[@]} ; do
> echo "The capital of $k is ${capitals[$k]}"
> done
The capital of Illinois is Springfield
The capital of California is SacramentoNice, never knew about this. Cheers
A nice article. Needs some apps. My suggestion is in the mid-late 90s I was making dough by writing CGI apps in BASH. Yes, BASH. lots of
#!/bin/bash
echo "Content-type: text/html"
echo
echo "<pre>"
(Run some CLI status type app here, or du -m a file or whatever)
echo "</pre>"
exit 0
Doing "web stuff" could be exciting for the learner.
(edited to add, doing "modern" web development in BASH might count as one of the counterexamples, no matter how fun and easy it is)
#!/bin/bash
echo "Content-type: text/html"
echo
echo "<pre>"
(Run some CLI status type app here, or du -m a file or whatever)
echo "</pre>"
exit 0
Doing "web stuff" could be exciting for the learner.
(edited to add, doing "modern" web development in BASH might count as one of the counterexamples, no matter how fun and easy it is)
This is a great idea.
Since I'm using these as lecture notes for my class, I'll try to add it in before I teach today.
Since I'm using these as lecture notes for my class, I'll try to add it in before I teach today.
Some fun ideas
If on a *nix platform try df for your cli app. This was "state of the art remote monitoring" where I worked in the mid 90s. Also http://whatever/cgi-bin/uptime displayed about what you'd expect. And the dmesg script, another handy one that is about what it sounded like.
Next step is there's a lot more HTML codes than <pre>. So put stuff in <h1> etc. Before <blink> was censored I used that occasionally.
A fun next step is commands with mysterious and unclear outputs. Like grep. Maybe grep a log file. Will you get no lines or a thousand lines? How do you intend to inform the user that either the CGI crashed or there is in fact no lines to output.
After that a bunch of if/then based on the output of grep -c
If its not fun to output text based on some if/then I'd try img tags. So now something broken is a red down arrow. And a giant frameset of multiple machines multiple tests is kind of an alert board, sort of. At least in 1997 or so.
You can make an interesting display of security problems by stashing temp files and temp scripts and temp things in /tmp... what could possibly go wrong with trusting the contents of something in /tmp, LOL?
Although I forget exactly how (its been awhile since BASH was a cutting edge web framework) there's a way to access http parameters so you can do simple forms that ask for something to search, then run grep on that passed parameter, then format the grep output to an attractive level. And of course you can pass "funny" parameters to the script which probably does no input checking and introduce them to Bobby Tables meme and all that.
I didn't do this, but in the years before phpmyadmin I worked at a place that basically had a script that ran whatever you passed it as a command for mysql. As root. With no authentication. And of course a hard coded password in the command line for anyone running "ps" to see. Yeah. It was one of those things that the developers thought was awesome for dev work on the test server but somehow snuck off the test server. That was a funny one, nothing bad happened, but could have been bad.
This kind of stuff is a bit safer on a private lan than being tested out on the internet, but everyone reading this probably knows that...
If on a *nix platform try df for your cli app. This was "state of the art remote monitoring" where I worked in the mid 90s. Also http://whatever/cgi-bin/uptime displayed about what you'd expect. And the dmesg script, another handy one that is about what it sounded like.
Next step is there's a lot more HTML codes than <pre>. So put stuff in <h1> etc. Before <blink> was censored I used that occasionally.
A fun next step is commands with mysterious and unclear outputs. Like grep. Maybe grep a log file. Will you get no lines or a thousand lines? How do you intend to inform the user that either the CGI crashed or there is in fact no lines to output.
After that a bunch of if/then based on the output of grep -c
If its not fun to output text based on some if/then I'd try img tags. So now something broken is a red down arrow. And a giant frameset of multiple machines multiple tests is kind of an alert board, sort of. At least in 1997 or so.
You can make an interesting display of security problems by stashing temp files and temp scripts and temp things in /tmp... what could possibly go wrong with trusting the contents of something in /tmp, LOL?
Although I forget exactly how (its been awhile since BASH was a cutting edge web framework) there's a way to access http parameters so you can do simple forms that ask for something to search, then run grep on that passed parameter, then format the grep output to an attractive level. And of course you can pass "funny" parameters to the script which probably does no input checking and introduce them to Bobby Tables meme and all that.
I didn't do this, but in the years before phpmyadmin I worked at a place that basically had a script that ran whatever you passed it as a command for mysql. As root. With no authentication. And of course a hard coded password in the command line for anyone running "ps" to see. Yeah. It was one of those things that the developers thought was awesome for dev work on the test server but somehow snuck off the test server. That was a funny one, nothing bad happened, but could have been bad.
This kind of stuff is a bit safer on a private lan than being tested out on the internet, but everyone reading this probably knows that...
[deleted]
I like how this article just presents you with the basics, but all that is really needed to get up to speed quickly and with little fuss.
Most bash tutorials I have seen seem to be taking forever to get to the parts I usually need a quick refresher of.
A very good quick start/reminder/advanced cheat sheet, good work.
Most bash tutorials I have seen seem to be taking forever to get to the parts I usually need a quick refresher of.
A very good quick start/reminder/advanced cheat sheet, good work.
Thanks!
I created these as the lecture notes for my class this morning.
I created these as the lecture notes for my class this morning.
Well, that was a good morning's work.
Is this markdown -> HTML or pandoc or your own static generator?
Is this markdown -> HTML or pandoc or your own static generator?
I write the HTML directly, and stitch it together with...bash scripts.
Should have guessed about the scripts :-)
This is why I prefer http://fishshell.com/ to bash. Bash just has too much noise in my opinion (like then keyword).
For simple ifs that don't require elses, I just use && or || as necessary:
test -e "$file" || {
echo "file not found: '$file'" >/dev/stderr
exit 1
}
See I think that there are at least two possible companion pieces to this article that I'd love to read. There are a lot of things in bash that aren't necessarily obviously useful to the novice bash programmer but become more important when you spend some time writing things, like $? for tests
Common use cases for some of the peculiar bash features with examples
historical explanation of how features developed were originally used
You may want to warn that your "convert a folder of images from JPG to PNG in parallel" might crash your machine. I didn't try it myself because I'd rather not have 230 convert processes running at once … https://www.gnu.org/software/parallel/ is the typical solution to this kind of task.
The linked relational shell programming article was really fun, informative, and inspiring.
Indispensible? What does it mean?
ed. "indispensable" got it.
ed. "indispensable" got it.
That said, I try not to do bash scripting. It's always been a boiling-the-frog thing for me: You start with a cute script with a couple of lines and then you make it more robust and add some more stuff and before you know, you've written a non-trivial program in an awfully quirky programming language.
But I'm still happy I've been there and know my way around bash fairly well. Makes for some pretty powerful quick one liners.