lhoursquentin

vor 4 Jahrendiscuss

The lack of arrays, dicts and local variables when trying to be POSIX compliant becomes quickly annoying when writing big programs though. There are of course workarounds to deal with those but I wish we didn't have to use them.

lhoursquentin

vor 4 Jahrendiscuss

What's interesting with this example of command1 | command2 is that some shells such as zsh will optimize the last member of the pipeline to be executed in the current process (nothing mandated by POSIX here), so effectively this works on zsh.

lhoursquentin

vor 4 Jahrendiscuss

> [[ ]] is a bash builtin, and is more powerful than [ ] or test.

Agreed on the powerful bit, however [[ ]] is not a "builtin" (whereas [ and test are builtins in bash), it's a reserved word which is more similar to if and while.

That why [[ ]] can break some rules that builtins cannot, such as `[[ 1 = 1 && 2 = 2 ]]` (vs `[ 1 = 1 ] && [ 2 = 2]` or `[ 1 = 1 -a 2 = 2 ]`, -a being deprecated).

Builtins should be considered as common commands (like ls or xargs) since they cannot bypass some fundamental shell parsing rules (assignment builtins being an exception), the main advantages of being a builtin being speed (no fork needed) and access to the current shell process env (e.g. read being able to assign a variable in the current process).

lhoursquentin

vor 4 Jahrendiscuss

> Yeah, I know there's D3.js but that involved way more knowledge and learning than I was interested in.

Same, d3 looked very powerful but had a steep learning curve. I was looking for something simple to generate process trees in real time and ended up using cytoscape js [0], helped me have a working POC in an hour, highly recommended.

[0] https://js.cytoscape.org/

lhoursquentin

vor 5 Jahrendiscuss

https://www.grymoire.com/Unix/Sed.html is a good resource to get started, though it's definitely something you should not just read due to the fact that sed code is pretty hard to read. Have a terminal on the side and try it out live. Once you understand pattern and hold spaces sed becomes a pretty fun esoteric language :)

lhoursquentin

vor 5 Jahrendiscuss

> I would be curious if mksh is also faster. ksh93 also claims great performance increases, but it is no longer (well) maintained.

mksh `printf` is not builtin, so if the script makes heavy use of it then it tends to be slower than bash.

ksh is very interesting because it has the ability to avoid spawning new processes on most command susbstitutions, making it usually much faster than all other POSIX shells.

But again this depends heavily on what the script being executed is doing, here's a highlight of what I described:

  $ time mksh -c 'for i in $(seq 1000); do printf $(printf $(printf $(printf hello))); done' > /dev/null
  
  real    0m2.121s
  user    0m1.321s
  sys     0m0.875s

  $ time ksh -c 'for i in $(seq 1000); do printf $(printf $(printf $(printf hello))); done' > /dev/null
  
  real    0m0.013s
  user    0m0.008s
  sys     0m0.005s

  $ time dash -c 'for i in $(seq 1000); do printf $(printf $(printf $(printf hello))); done' > /dev/null
  
  real    0m0.311s
  user    0m0.279s
  sys     0m0.073s
  
  $ time bash -c 'for i in $(seq 1000); do printf $(printf $(printf $(printf hello))); done' > /dev/null
  
  real    0m0.664s
  user    0m0.538s
  sys     0m0.186s

lhoursquentin

vor 6 Jahrendiscuss

Cool I'll definitely try to set it up in the coming days!

Here's my humble strace visualizer: https://lhoursquentin.github.io/visual-strace/

lhoursquentin

vor 6 Jahrendiscuss

Great post, also love what you are trying to do with C playground, this is awesome!

I've recently been trying to build something similar, visualizing forks/exeve/read/write, but using the strace output of a binary, which is much less challenging.

lhoursquentin

vor 6 Jahrendiscuss

You should probably note that this hack spawns an additional shell process per comment :)

lhoursquentin

vor 6 Jahrendiscuss

Yes most of the time the respecting POSIX to the letter is not needed, but it is of course satisfying knowing that your script can run fine on BSDs and other less common distributions :)

Though sometimes you don't need to go that far to break stuff, for instance switching from Fedora to Ubuntu. I've seen many scripts fail on debian derivatives because people think using #!/bin/sh as a shebang is fine since it works on their computer where sh was in fact a symlink to bash.

But on debian based distributions /bin/sh is often dash, not bash, and dash is basically the strict POSIX subset + local, all fancy stuff like [[ ]], &>, arrays, ... will fail.

Though this is less about long options here and more about general shell scripting.

lhoursquentin

vor 6 Jahrendiscuss

> Remember readable means that you can read and understand it

Yes you are right, my usage of "readable" was probably wrong there.

Maybe the point I was trying to make was less about the individual constructs but more about how the lack of "common" features makes the whole program less "understandable", or maybe less easy to get familiar with, simply due to the amount of code needed to achieve a specific task.

The same way assembly is considered less "readable" than C. Not because assembly is less readable on a line by line basis, it's even simpler, but because of the number of lines and operations needed to achieve a simple task.

Basically it's easier to understand 10 lines than a 1000.

lhoursquentin

vor 6 Jahrendiscuss

The thing is when you restrict yourself to only what POSIX specifies you don't even have basic data structures, no array, no hash map not even local variables, and that often leads to code that relies on multiple hacks.

For instance: you can't even slice "$@" which is the only array like construct specified by POSIX (excluding unsafe splitting), so you end up shifting and re-enqueuing with set -- "$@" "$1", which is unreadable.

Want to read a single character? Impossible with the POSIX read, but you can workaround with dd

etc.

All those workarounds to POSIX limitations are fascinating, but it forces a lot of arcane constructs, that's for sure.

And of course this is usually the point where you ask yourself if you've chosen the correct language for the task, but that's another debate.

lhoursquentin

vor 6 Jahrendiscuss

-i is definitely not specificed by POSIX, but it supported on all those platforms with some small differences, for instance on OSX the backup extension (-i.bak) is not optional.

lhoursquentin

vor 6 Jahrendiscuss

Yep I can confirm intel cards are still an issue. I installed FreeBSD a week ago and gave up trying to enable Wi-Fi after an hour of struggle on a Thinkpad T490, didn't even get to the point of having the interface show up in ifconfig. Ended up using my phone in USB tethering mode instead.

lhoursquentin

vor 6 Jahrendiscuss

Yep that's exactly right :) !

To clarify my statement POSIX requires at least 8192 bytes for the pattern and hold space, and I chose to allow at most 8192 bytes here, which indeed means that this will prevent working on really long lines or storing entire files in the hold space for instance.

lhoursquentin

vor 6 Jahrendiscuss

No concrete benchmarks no, I just timed a few scripts I had on hand and generated a bunch of others, and I saw compiled ones performed slightly faster than GNU and toybox sed, so nothing serious.

Unfortunately, it's also pretty hard to find big POSIX sed scripts in the wild, so my speed observations are centered around my own scripts. I would definitely be interested in learning more about sed scripts taking hours to run though, if you have something that I could check out that would be awesome!

And talking about speed, I think there's also a small margin for improvement in this project, like avoiding compiling the same regex multiple times (if it appears in different places in the script), and some places that could probably benefit from using hash tables instead of static arrays (address ranges for instance). More work could be done on the translation side regarding backrefs, which are parsed on the C side for now.

lhoursquentin

vor 6 Jahrendiscuss

A Sed to C translator written in Sed, the objective being to be able to translate itself: https://github.com/lhoursquentin/sed-bin

lhoursquentin

vor 6 Jahrendiscuss

Really cool stuff, I've been working on being able to debug sed scripts as well by translating it to pseudo readable C[1], compiling the result and then using gdb to step in the script, but this TUI is much more readable!

[1] https://github.com/lhoursquentin/sed-bin

lhoursquentin

vor 6 Jahrendiscuss

Regarding 4. it's an interesting point, and this actually exists, few people know that ksh93 (which was a big inspiration for bash and zsh) does not fork when it doesn't really need to, for instance if you strace the following:

echo $(echo $(echo echo))

Due to those command substitutions, on zsh/bash/mksh/dash/yash/busybox/osh you'll see 2 forks/clones, but on ksh93 there will be none, and this allows ksh93 to be insanely fast on "pure" shell scripts that have little use of external binaries and pipes, even faster than dash which is usually known to outperform most shells.

lhoursquentin

Submissions

Show HN: Visual strace – visualize processes forks and pipes

Show HN: Portable object-oriented shell scripting

Show HN: Sed to C translator written in sed

comments