The Awk state machine parser pattern (2018)(two-wrongs.com)
two-wrongs.com
The Awk state machine parser pattern (2018)
https://two-wrongs.com/awk-state-machine-parser-pattern.html
10 comments
This is basically what Perl used to be used for too.
If you think of Ruby as a more readable / maintainable Perl -- it's much better suited to these text processing tasks.
Ruby even supports Perl regular expressions which are more powerful and convenient than Awk's.
Some version of Ruby is usually in the base system of every Linux system (perl5 is more ubiquitous but much more cryptic)
Ruby even supports Perl regular expressions which are more powerful and convenient than Awk's.
Some version of Ruby is usually in the base system of every Linux system (perl5 is more ubiquitous but much more cryptic)
>Ruby even supports Perl regular expressions
No, Ruby Regexp is based on the https://github.com/k-takata/Onigmo library. There are plenty of differences compared to Perl, for example `^` and `$` anchors always match start/end of lines without needing a flag, subexpression syntax uses `\g` instead of `(?N)` and so on.
No, Ruby Regexp is based on the https://github.com/k-takata/Onigmo library. There are plenty of differences compared to Perl, for example `^` and `$` anchors always match start/end of lines without needing a flag, subexpression syntax uses `\g` instead of `(?N)` and so on.
I'm a bit upset to say that this is one of the few times I've seen AWK code outside of a one liner (some of those one liners are pretty beastly, but still).
It reads pretty well, and now I'm interested in using it a bit more for my scripts. Any good AWK examples/resources anyone can recommend?
It reads pretty well, and now I'm interested in using it a bit more for my scripts. Any good AWK examples/resources anyone can recommend?
The AWK book written by the A W K from the program name has always been considered the bible.
Edit: misattributed to k&r
Edit: misattributed to k&r
This one is good; clear explanations of the concepts and various useful examples to crib from: https://www.grymoire.com/Unix/Awk.html
Their document covering sed is also excellent.
Their document covering sed is also excellent.
AWK supports conditional branching and switching so you can represent nested states as well. Wouldn't recommended beyond depth ~1 though... Use a proper language for that.
[deleted]
TXR:
@(bind idtab @(relate '("radeon" "k10temp")
'("GPU" "CPU")
"SYS"))
@(bind thash @(hash))
@(repeat)
@id-@bus-@code
Adapter: @name
@ (repeat)
temp@num: @{temp}°C@nil
@ (do (set [thash `@[idtab id]_@num`] temp))
@ (until)
@ (end)
@(end)
@(do (dohash (tag temp thash)
(sh `echo gmetric -t uint16 -u Celsius -n @tag -v @temp`)))
$ txr temp.txr temp.dat
gmetric -t uint16 -u Celsius -n SYS_2 -v +43.0
gmetric -t uint16 -u Celsius -n SYS_1 -v +31.0
gmetric -t uint16 -u Celsius -n CPU_1 -v +36.8
gmetric -t uint16 -u Celsius -n GPU_1 -v +50.5
gmetric -t uint16 -u Celsius -n SYS_3 -v +38.0
$ gawk -f temp.awk temp.dat
gawk: temp.awk:32: temp = substr(
gawk: temp.awk:32: ^ unexpected newline or end of string
gawk: temp.awk:35: matches[1, "length"]
gawk: temp.awk:35: ^ unexpected newline or end of string
gawk: temp.awk:35: );
gawk: temp.awk:35: ^ 0 is invalid as number of arguments for substr
$ mawk -f temp.awk temp.dat
mawk: temp.awk: line 30: regular expression compile failed (missing operand)
+([0-9.]+)°C
mawk: temp.awk: line 30: syntax error at or near ,
mawk: temp.awk: line 31: missing ) near end of line
mawk: temp.awk: line 32: syntax error at or near ,
mawk: temp.awk: line 35: extra ')'
The blog's code works (with gawk) if some whitespace errors are fixed.
Awk has some newline sensitivities due to the following ambiguities:
And thus (IIRC) the awk lexical analyzer (in the original One True Awk implementation) returns an explicit newline token to the Yacc parser. In any phrase structure that doesn't deal with that token, a newline will cause a syntax error:
The newline sensitivities don't seem to serve a purpose in the C-like language within the actions.
That language also occurs outside of actions via the function construct:
$ gawk -f temp.awk temp.dat # echo inserted into command
gmetric -t uint16 -u Celsius -n GPU_1 -v 50.5
gmetric -t uint16 -u Celsius -n SYS_1 -v 31.0
gmetric -t uint16 -u Celsius -n SYS_2 -v 43.0
gmetric -t uint16 -u Celsius -n SYS_3 -v 38.0
gmetric -t uint16 -u Celsius -n CPU_1 -v 36.8
Awk has some newline sensitivities due to the following ambiguities:
condition # condition with no action allowed: default { print } action
{ action } # action with no condition allowed
condition { action } # both
condition
{ action }
And thus (IIRC) the awk lexical analyzer (in the original One True Awk implementation) returns an explicit newline token to the Yacc parser. In any phrase structure that doesn't deal with that token, a newline will cause a syntax error:
function( # no good
arg
)
function("string " # no good
foo + bar
" catenation")
The newline sensitivities don't seem to serve a purpose in the C-like language within the actions.
That language also occurs outside of actions via the function construct:
function whatever(...) {
}
My first thought when I saw this item in the HN list was TXR :)
Here is the data into JSON (keeping the values a
$ txr json.txr temp.dat
{"radeon-pci-0100":{"temp1":{"crit":120,"hyst":90,"value":50.5}},
"f71889ed-isa-0480":{"temp2":{"hyst":77,"value":43,"high":85},"alarm":{"fan3":true,"fan2":true},
"sensor":{"crit":100,"name":"thermistor","hyst":92},"max-voltage":{"in1":2.04},
"temp1":{"hyst":81,"value":31,"high":85},"voltage":{"in5":1.23,"+3.3V":3.23,"in6":1.53,"in2":1.09,"Vbat":3.31,"in1":1.07,
"in4":0.58,"3VSB":3.25,"in3":0.89},
"temp3":{"hyst":68,"value":38,"high":70},"rpm":{"fan3":0,"fan1":3978,"fan2":0}},
"k10temp-pci-00c3":{"temp1":{"crit":80,"hyst":78,"value":36.8,"high":70}}}
@(bind dict @(hash))
@(name file)
@(repeat)
@idstring
Adapter: @adapter
@ (collect :vars (entry))
@ (line line)
@ (assert error `unhandled stuff occurs at @file:@line`)
@ (some)
@{temp /temp\d+/}: @{val}°C (crit = @{crit}°C,
hyst = @{hyst}°C)
@ (bind entry @#J^{~temp : { "value" : ~val,
"crit" : ~crit,
"hyst" : ~hyst }})
@ (or)
@{temp /temp\d+/}: @{val}°C (high = @{high}°C)
(crit = @{crit}°C,
hyst = @{hyst}°C)
@ (bind entry @#J^{~temp : { "value" : ~val,
"crit" : ~crit,
"hyst" : ~hyst,
"high" : ~high }})
@ (or)
@{temp /temp\d+/}: @{val}°C (high = @{high}°C,
hyst = @{hyst}°C)
@ (bind entry @#J^{~temp : { "value" : ~val,
"high" : ~high,
"hyst" : ~hyst }})
@ (or)
(crit = @{crit}°C,
hyst = @{hyst}°C)
sensor = @sensor
@ (bind entry @#J^{"sensor" : { "name" : ~sensor,
"crit" : ~crit,
"hyst" : ~hyst }})
@ (or)
@label: @voltage V
@ (bind entry @#J^{"voltage" : {~label : ~voltage}})
@ (or)
@label: @voltage V (max = @max V)
@ (bind entry @#J^{"voltage" : {~label : ~voltage},
"max-voltage" : {~label : ~max}})
@ (or)
@label: @rpm RPM
@ (bind entry @#J^{"rpm" : {~label : ~rpm}})
@ (or)
@label: @rpm RPM ALARM
@ (bind entry @#J^{"rpm" : {~label : ~rpm},
"alarm" : {~label : true}})
@ (or)
@ (bind entry @#J{})
@ (end)
@ (until)
@ (end)
@ (do (set [dict idstring]
(reduce-left (op hash-uni @1 @2 hash-uni) entry #J{})))
@(end)
@(do
(defun numify (dict)
(dohash (k v dict dict)
(typecase v
(string (iflet ((f (tofloat v)))
(set [dict k] f)))
(hash (numify v)))))
(put-jsonl (numify dict)))State machine of the form
Here's a chapter from my GNU awk one-liners book with more such examples: https://learnbyexample.github.io/learn_gnuawk/processing-mul...
awk '/start/{f=1} f; /end/{f=0}'
Here's a chapter from my GNU awk one-liners book with more such examples: https://learnbyexample.github.io/learn_gnuawk/processing-mul...
I have to say, that's the most readable and understandable Awk program I've seen.
Does anyone know if there's a repository of similarly literate awk scripts?
Does anyone know if there's a repository of similarly literate awk scripts?
I’ve found that AWK is frequently surprisingly readable actually, as long as you understand the execution model. I think people tend to think of it (with some justification) as in the same vein as Perl, but it isn’t nearly as surface-level cryptic. The syntax is just “C-style” with dynamic typing and built in regular expressions.
I have a couple of AWK scripts to handle my personal finances (basically, consuming various bank/credit card statements and turning them to Ledger files) and it’s just the perfect language for that kind of task. My scripts look fairly similar to the examples in the blog post, they also use the same state-machine trick. With the possible exception of Perl/Raku, it’s my favourite language for that kind of thing.
I have a couple of AWK scripts to handle my personal finances (basically, consuming various bank/credit card statements and turning them to Ledger files) and it’s just the perfect language for that kind of task. My scripts look fairly similar to the examples in the blog post, they also use the same state-machine trick. With the possible exception of Perl/Raku, it’s my favourite language for that kind of thing.
Have a look at the classic "The AWK Programming Language" by the A, W, and K in awk. It's full of great examples.
https://ia803404.us.archive.org/0/items/pdfy-MgN0H1joIoDVoIC...
https://ia803404.us.archive.org/0/items/pdfy-MgN0H1joIoDVoIC...
There used to be very good examples in awk.info. The domain is on sale now, but you can get all the old content from archive.org and it still is very valid.
I don't know if this is idiomatic way of doing awk. It is just a port of a python script to awk.
https://github.com/berry-thawson/diff2html
This is first attempt in writing awk script. Would like to know how readable it is.
Edits : added a new line. changed some words.
https://github.com/berry-thawson/diff2html
This is first attempt in writing awk script. Would like to know how readable it is.
Edits : added a new line. changed some words.
Not to disparage the nice awk script, but reading from /sys/class/hwmon/* seems more sensible...
(Which is my way of saying, rather than writing a script like this, I'd spend some time to get the data machine readable in the first place — or even just dig up where to already find it in a machine readable form.)
(Which is my way of saying, rather than writing a script like this, I'd spend some time to get the data machine readable in the first place — or even just dig up where to already find it in a machine readable form.)
Reading a file from sysfs is great and script-friendly, sure. OTOH, finding the right file to read is less straight-forward.
For one, depending on kernel version and compile options, temperature/voltage/rpm files could be found under /sys/class/hwmon, or /sys/devices/virtual, or /sys/devices/platform/soc. And then, say, script found a dozen of those "temp", or "temp∗_input", or "microvolts" files. How to figure out which one is for CPU, motherboard, battery, PSU, air intake? Probably with extra logic, reading corresponding "temp∗_label", if those even exist? Parsing /sys/firmware/devicetree ? Taking hints from parts of the path-name, where such files are found?
lm_sensors is no silver bullet here either, but at least it does passable job discovering/labeling sensors most of the time.
For one, depending on kernel version and compile options, temperature/voltage/rpm files could be found under /sys/class/hwmon, or /sys/devices/virtual, or /sys/devices/platform/soc. And then, say, script found a dozen of those "temp", or "temp∗_input", or "microvolts" files. How to figure out which one is for CPU, motherboard, battery, PSU, air intake? Probably with extra logic, reading corresponding "temp∗_label", if those even exist? Parsing /sys/firmware/devicetree ? Taking hints from parts of the path-name, where such files are found?
lm_sensors is no silver bullet here either, but at least it does passable job discovering/labeling sensors most of the time.
Well, in that case:
# sensors --help
Usage: sensors [OPTION]... [CHIP]...
…
-j Json outputI used a similar technique for my awk markdown parser: https://github.com/yiyus/md2html.awk
An awk state machine is a quite straightforward way to deal with data like this log file. It is not so clear that this is the best way to write a relatively large piece of software, like a markdown parser (when I wrote md2html.awk in 2009, the standard md parser was the original one by John Gruber, written in Perl, so it actually was an improvement in code clarity, performance, and portability (we had no perl in Plan 9!), but nowadays it is easy to find much better solutions).
An awk state machine is a quite straightforward way to deal with data like this log file. It is not so clear that this is the best way to write a relatively large piece of software, like a markdown parser (when I wrote md2html.awk in 2009, the standard md parser was the original one by John Gruber, written in Perl, so it actually was an improvement in code clarity, performance, and portability (we had no perl in Plan 9!), but nowadays it is easy to find much better solutions).
Wow, I've bookmarked this.
I also wrote an Markdown to HTML converter in Awk once, though my purpose was to convert marked down comments in source code to documentation. I started with an Awk script to extract the comments and then systematically added markdown features. My end result isn't very elegant though.
https://github.com/wernsey/d.awk/blob/master/d.awk
I also wrote an Markdown to HTML converter in Awk once, though my purpose was to convert marked down comments in source code to documentation. I started with an Awk script to extract the comments and then systematically added markdown features. My end result isn't very elegant though.
https://github.com/wernsey/d.awk/blob/master/d.awk
Glad I’m not the only one crazy enough to write long AWK parsers. Here’s my tool to convert “feature tables” into “bed” files, both files that describe genomes, but for some reason, NCBI uses the former, even though it’s totally useless. https://github.com/ryandward/tbl2bed/blob/main/tbl2bed.awk