Bash-ini-parser: Advanced bash INI parser library(github.com)
github.com
Bash-ini-parser: Advanced bash INI parser library
https://github.com/lsferreira42/bash-ini-parser
7 comments
Long before systemd took over the init world, Debian led a huge effort to switch their entire infrastructure to dash[1], specifically making heavy use of checkbashisms[2]. The main reason behind it was performance: POSIX sh implementations were much simpler, and easier to optimise.
[1]: http://gondor.apana.org.au/~herbert/dash/
[2]: https://linux.die.net/man/1/checkbashisms
My rule of thumb is that if a shell script calls for a Bash-only feature, exceeds 100 lines, defines more than three functions, includes a nested loop, or a pipeline that exceeds 80 characters, it's probably time to reach for Python.
There are reasonable cases for exceptions - my backup script has about 200 lines and 6 functions, but it's 100% portable between Linux & macOS.
[1]: http://gondor.apana.org.au/~herbert/dash/
[2]: https://linux.die.net/man/1/checkbashisms
My rule of thumb is that if a shell script calls for a Bash-only feature, exceeds 100 lines, defines more than three functions, includes a nested loop, or a pipeline that exceeds 80 characters, it's probably time to reach for Python.
There are reasonable cases for exceptions - my backup script has about 200 lines and 6 functions, but it's 100% portable between Linux & macOS.
> Using pip-installed libraries brings in the famous packaging problems, but you don't have to use it!
These days, with uv and PEP 723 inline script metadata, that's not an issue either.
These days, with uv and PEP 723 inline script metadata, that's not an issue either.
# /// script
# dependencies = ["numpy"]
# requires-python = ">=3.12"
# ///
import numpy as np
uv run myscript.py
And if you come back to this script in a few years time and it pulls a newer version of numpy with an incompatible api, there is no easy way of knowing which version it was designed to be used with...
You can of course absolutely use `"numpy~=1.12"` as a requirement.
Only if you didn't run `uv lock --script` [0] and commit the lockfile. If you don't want that second file sitting next to the script, you could instead specify an `exclude-newer` timestamp, and uv won't resolve any dependency versions newer than that.
It might be cool if uv ignored dependency versions newer than the script's last modified timestamp, but this behavior would probably need to be very explicit to avoid the situation where a working script is mysteriously broken by making an innocuous change which updates its last modified timestamp, causing uv to resolve a newer, incompatible dependency version.
[0] https://docs.astral.sh/uv/guides/scripts/#locking-dependenci...
It might be cool if uv ignored dependency versions newer than the script's last modified timestamp, but this behavior would probably need to be very explicit to avoid the situation where a working script is mysteriously broken by making an innocuous change which updates its last modified timestamp, causing uv to resolve a newer, incompatible dependency version.
[0] https://docs.astral.sh/uv/guides/scripts/#locking-dependenci...
Have you seen the x-cmd package? It's nuts
https://x-cmd.com
It's legitimately impressive, the person/people behind this wrote some pretty interesting code, I've learned some things about Bash reading through parts of the code. But it's crazy at the same time.
https://x-cmd.com
It's legitimately impressive, the person/people behind this wrote some pretty interesting code, I've learned some things about Bash reading through parts of the code. But it's crazy at the same time.
People are generally bad about following their own advice. They're worst at following advice to avoid shell scripts. :-) We all know in theory we should move things to Python or another robust language once they become sufficiently complex, but in the moment, adding just this one feature takes precedence. Iterate "just one feature" a few times and you end up with a 1,000-line bash program. (The word "script" becomes inapplicable.) Now what? You want to add configuration support to your bash program? Well, you might as well use something robust to do it.
Which is why you rewrite in Python long before reaching that threshold. I start considering it as early as when I need to add any non-trivial command line parsing: argparse is just so much better than "while getopt".
This is why my threshold for not using any shell is "does it have more than one loop, more than one if-statement, an argument or file format parser or more than 10 lines?". Meaning that only very trivial things can be shell scripts and you rewrite early enough such that it isn't really a hassle or painful.
It does not have to be all-at-once.
Your bash program takes 10 input args and requires 15 env variables to be set? Don't reach out for that bash ini parser, create a python script which parses configs and invokes the original bash script. Next time you change feature, move it to python. Eventually you'll be mostly python.
Something that helps greatly with that is that you can embed bash in python. So initial conversion can be as simple as file rename + 3 lines of code.
Your bash program takes 10 input args and requires 15 env variables to be set? Don't reach out for that bash ini parser, create a python script which parses configs and invokes the original bash script. Next time you change feature, move it to python. Eventually you'll be mostly python.
Something that helps greatly with that is that you can embed bash in python. So initial conversion can be as simple as file rename + 3 lines of code.
import subprocess
subprocess.run(["bash", "-c", """
echo This is bash
if hostname | grep box; then
echo running on a box
fi
"""], check=True)
It's not that simple. How are you passing argv to the bash script? What about how `subprocess.run` changes the error output to include Python exception spam? Plus, in most editors, you lose all the bash syntax highlighting in the bash program chunk. I haven't seen a lot of hybrid programs like this work well.
That's why it's just the first stage of an incremental rewrite.
argv is easy to pass, though.
argv is easy to pass, though.
It is that simple. You can pass arguments to `bash -c` with no problems(just don't forget a dummy $0). If you don't want exception spam, replace `check=True` with explicit exit. Here you go, it's 5 lines now.
The bash syntax highlighting will disappear, correct, and that is a downside. But hey, you are supposed to be rewriting this in python, so it's just more motivation :)
#!/usr/bin/env python3
import sys
import subprocess
result = subprocess.run(["bash", "-c", """
echo I am in bash, I have $# args, arg1 is "$1"
set -x
"$@"
""", "dummy.sh"] + sys.argv[1:])
sys.exit(result.returncode)
subprocess.run(["bash", "-c", f"rsync {shlex.quote(filename)} somehost: | some-script"])
The bash syntax highlighting will disappear, correct, and that is a downside. But hey, you are supposed to be rewriting this in python, so it's just more motivation :)
> And if your system has bash, it often has python as well, or at least it's easy to install.
The problem is, bash is a universal target. Every unixoid system has bash in one way or another, usually at /bin/bash or in a pinch /usr/bin/env bash. No matter if your target system is Debian oldoldstable or bleeding edge Gentoo, it will run.
Python however is a hot mess. Some operating systems are at `python` yielding v2, others v3. Others (Macports) have python2 and python3 binaries if the user doesn't also do a `port select` afterwards. The user may have overridden this with virtualenv's (that this exists at all is a disgrace in itself). And that all is before `pip` even has entered the stage, or before you try to figure out what whitespace format Python expects and what whitespace format you just used in the last YAML file you edited.
(The whitespace shit is what will keep me from using Python if at all possible.)
The problem is, bash is a universal target. Every unixoid system has bash in one way or another, usually at /bin/bash or in a pinch /usr/bin/env bash. No matter if your target system is Debian oldoldstable or bleeding edge Gentoo, it will run.
Python however is a hot mess. Some operating systems are at `python` yielding v2, others v3. Others (Macports) have python2 and python3 binaries if the user doesn't also do a `port select` afterwards. The user may have overridden this with virtualenv's (that this exists at all is a disgrace in itself). And that all is before `pip` even has entered the stage, or before you try to figure out what whitespace format Python expects and what whitespace format you just used in the last YAML file you edited.
(The whitespace shit is what will keep me from using Python if at all possible.)
That is why you don't use Python, you use Perl. Every Unix has Perl, and Perl has backward compatibility into the last millenium. Always use 'strict' and proper linting for readability and typo avoidance.
If you are able to install modules, add 'IPC::Run' and 'Getopt::Declare' or 'Getopt::Euclid'. Never again will you be able to tolerate anything else ;)
If you are able to install modules, add 'IPC::Run' and 'Getopt::Declare' or 'Getopt::Euclid'. Never again will you be able to tolerate anything else ;)
Yeah, the naming was pretty bad for a while, but it seems today `#!/usr/bin/env python3` is a safe bet. And pip/virtualenv should not matter - remember, we've are talking about "python-without-dependencies"
As for tabs vs spaces - I really don't get it... My editor (emacs) picks the style automatically, I don't think I even had to set this up. But most of the python code I see use 2 or 4 spaces, so if someone puts tabs, it's immediately visible 8-character indents, which makes me alert (and then I turn on whitespace highlighting in my editor).
As for tabs vs spaces - I really don't get it... My editor (emacs) picks the style automatically, I don't think I even had to set this up. But most of the python code I see use 2 or 4 spaces, so if someone puts tabs, it's immediately visible 8-character indents, which makes me alert (and then I turn on whitespace highlighting in my editor).
uv has pretty much solved those issues, but it does need to be installed. Fortunately it's very easy to install.
Another option I've used is Deno.
Another option I've used is Deno.
That's all well and good, but if you're looking for a language that will survive for decades, then BASH or shell scripts are a better choice. Often, there'll be some ancient machine that's been almost forgotten about and will die and need replacing. I can trivially copy a 30-year old BASH script and run it on a new machine, but can you do that with a 30-year old Python programme? How will today's Python survive the next 30 years?
python 3.0 code, released back in 2008, is still compatible with latest released python, with some exceptions like asyncio or obscure file formats which are unlikely to appear in scripts. The pip packages likely did break, but we are talking about python-with-stdlib here.
Granted, it's not 30 years old, but 17 years is pretty impressive. And I don't think there will be another big break, the 2.0->3.0 transition was so painful they'll likely avoid another one for decades.
Granted, it's not 30 years old, but 17 years is pretty impressive. And I don't think there will be another big break, the 2.0->3.0 transition was so painful they'll likely avoid another one for decades.
17 years is very good. I don't have anything against Python, but there's a particular niche that shell/BASH scripts are ideal for. I keep considering writing more stuff in Python, but BASH operates at just the right level for tying systems together (computer duck tape).
> rewrite it in a real language
This refrain shows up in every shell script discussion here, and hot take, I find it mostly ill-founded hogwash. Some of the worst spaghetti code I've seen is such Python and Javascript rewrites.
Choice of language only really helps when it encourages developers to use patterns that fit the underlying problem domain. However, Python's path of least resistance is OOP architecture, which usually fits terribly in the domains where shell scripts are ubiquitous, e.g. glue code between systems, infra administrative tasks, etc.
Almost all shell script tasks boil down to executing a linear collection of steps, and by organizing code thusly, I find shell scripts to keep code short, sweet, and very maintainable. Using a data-first design, with well-designed global state typically meshes well with shell's strengths—dynamic scope, pipes, word-splitting, etc, not to mention coreutils, grep, awk, etc.
IMHO, all the shell hate boils down to not recognizing an impedance mismatch between the programming paradigms of the programmer and language. Try writing Haskell like you would write Java or vice-vesa, and you'll end up with just as much of a mess.
</rant> This topic is a bug-bear of mine, for whatever reason. :shrug:
This refrain shows up in every shell script discussion here, and hot take, I find it mostly ill-founded hogwash. Some of the worst spaghetti code I've seen is such Python and Javascript rewrites.
Choice of language only really helps when it encourages developers to use patterns that fit the underlying problem domain. However, Python's path of least resistance is OOP architecture, which usually fits terribly in the domains where shell scripts are ubiquitous, e.g. glue code between systems, infra administrative tasks, etc.
Almost all shell script tasks boil down to executing a linear collection of steps, and by organizing code thusly, I find shell scripts to keep code short, sweet, and very maintainable. Using a data-first design, with well-designed global state typically meshes well with shell's strengths—dynamic scope, pipes, word-splitting, etc, not to mention coreutils, grep, awk, etc.
IMHO, all the shell hate boils down to not recognizing an impedance mismatch between the programming paradigms of the programmer and language. Try writing Haskell like you would write Java or vice-vesa, and you'll end up with just as much of a mess.
</rant> This topic is a bug-bear of mine, for whatever reason. :shrug:
I used to do the same but lately I've been enjoying writing scripts using bun.js and its shell API: https://bun.sh/docs/runtime/shell
The main reason is piping commands, way easier than python. It's very easy to convert a bash script to bun.
The main reason is piping commands, way easier than python. It's very easy to convert a bash script to bun.
I agree with you on that, but I had this task on a legacy project with over 50k lines of shell scripts, and out of nowhere, it needed to start reading INI files and loading those values into environment variables. That's why this was written!
> My rule has always been that once a shell script gets too complex, it's time to rewrite it in real language
Is Powershell a real language? Because it is a shell script too.
I think you meant to say "data types" and "data structures"?
Is Powershell a real language? Because it is a shell script too.
I think you meant to say "data types" and "data structures"?
For similar functionality have a look at crudini,
which is a utility designed for interacting with ini files from shell scripts:
https://github.com/pixelb/crudini
https://github.com/pixelb/crudini
Hint: for cicd usage you can parse ini files with git, often present in such situations.
OT:
There is a initiative around bringing Indian languages into today's tech.
Named BHASHINI.
I initially thought the title was referring to that project given the similarity in name.
(Bhasha = Language)
There is a initiative around bringing Indian languages into today's tech.
Named BHASHINI.
I initially thought the title was referring to that project given the similarity in name.
(Bhasha = Language)
An ini->JSON converter + jq should be enough.
For ingestion, yes, but if you want to write back the ini, you have now lost all comments. Besides, jq's syntax is ... arcane.
You get to choose the JSON schema for ini, and so you can choose to preserve comments. As to jq as a language, if you get used to it it's really awesome.
Definitely one of those "if you're using it you've made a big mistake somewhere" pieces of software.
My recommendation for rewrite is "python with only stdlib". Using pip-installed libraries brings in the famous packaging problems, but you don't have to use it! Python's stdlib is fully "batteries included", and it has lots of useful things like ini file parser. And if your system has bash, it often has python as well, or at least it's easy to install. Things might get a bit more verbose, but they would be much more reliable and debuggable.