kbrazil · HackerTrans

kbrazil

no profile record

Submissions

Tips on adding JSON output to your CLI app

blog.kellybrazil.com

183 pointsby kbrazil5년 전110 comments

comments

3년 전discuss

There could be a major schema change that breaks the contract, but one of the nice things about JSON output is that it allows the creation of new fields without affecting downstream consumers.

That is, if I have a CLI program that spits out a list of IP addresses and one day I want to also output the corresponding dns names, I can simply add the "dns" field and existing pipelines will ignore the field and work just fine.

This is better than grep/awking/etc. unstructured text to STDOUT because, depending on how the author decides to add the new field, it can easily break existing pipelines that rely on the shape of the data to stay the same.

3년 전discuss

`jc` supports plugins:

https://github.com/kellyjonbrazil/jc#parser-plugins

3년 전discuss

`jc` author here. I've been maintaining `jc` for nearly four years now. Most of the maintenance is choosing which new parsers to include. Old parsers don't seem to have too many problems (see the Github issues) and bugs are typically just corner cases that can be quickly addressed along with added tests. In fact there is a plugin architecture that allows users to get a quick fix so they don't need to wait for the next release for the fix. In practice it has worked out pretty well.

Most of the commands are pretty old and do not change anymore. Many parsers are not even commands but standard filetypes (YAML, CSV, XML, INI, X509 certs, JWT, etc.) and string types (IP addresses, URLs, email addresses, datetimes, etc.) which don't change or use standard libraries to parse.

Additionally, I get a lot of support from the community. Many new parsers are written and maintained by others, which spreads the load and accelerates development.

3년 전discuss

Also, `jc` automatically selects the correct /proc/file parser so you can just do `jc /proc/meminfo` or `cat /proc/meminfo | jc --proc` without specifying the actual proc parser (though you can do that if you want)

Disclaimer: I'm the author of `jc`.

3년 전discuss

You can do it like this with Jello (I am the author):

    jello '[e.commit for e in _ if e.commit.author == "Tom Hudson"]'

Jello let’s you use python syntax with dot notation without the stdin/stdout/json.loads boilerplate.

https://github.com/kellyjonbrazil/jello

4년 전discuss

Interesting - the proc-pid-parser actually parses that file name just fine:

    $ echo '2001 (bad) S 1 2 3 4 5) S 1888 2001 1888 34816 2001 4202496 428 0 0 0 0 0 0 0 20 0 1 0 75513 115900416 297 18446744073709551615 4194304 5100612 140737020052256 140737020050904 140096699233308 0 65536 4 65538 18446744072034584486 0 0 17 0 0 0 0 0 0 7200240 7236240 35389440 140737020057179 140737020057223 140737020057223 140737020059606 0' | jc --proc-pid-stat

    {"pid":2001,"comm":"bad) S 1 2 3 4 5","state":"S","ppid":1888,"pgrp":2001,"session":1888,"tty_nr":34816,"tpg_id":2001,"flags":4202496,"minflt":428,"cminflt":0,"majflt":0,"cmajflt":0,"utime":0,"stime":0,"cutime":0,"cstime":0,"priority":20,"nice":0,"num_threads":1,"itrealvalue":0,"starttime":75513,"vsize":115900416,"rss":297,"rsslim":18446744073709551615,"startcode":4194304,"endcode":5100612,"startstack":140737020052256,"kstkeep":140737020050904,"kstkeip":140096699233308,"signal":0,"blocked":65536,"sigignore":4,"sigcatch":65538,"wchan":18446744072034584486,"nswap":0,"cnswap":0,"exit_signal":17,"processor":0,"rt_priority":0,"policy":0,"delayacct_blkio_ticks":0,"guest_time":0,"cguest_time":0,"start_data":7200240,"end_data":7236240,"start_brk":35389440,"arg_start":140737020057179,"arg_end":140737020057223,"env_start":140737020057223,"env_end":140737020059606,"exit_code":0,"state_pretty":"Sleeping in an interruptible wait"}

But the "magic" signature doesn't recognize it:

    $ echo '2001 (bad) S 1 2 3 4 5) S 1888 2001 1888 34816 2001 4202496 428 0 0 0 0 0 0 0 20 0 1 0 75513 115900416 297 18446744073709551615 4194304 5100612 140737020052256 140737020050904 140096699233308 0 65536 4 65538 18446744072034584486 0 0 17 0 0 0 0 0 0 7200240 7236240 35389440 140737020057179 140737020057223 140737020057223 140737020059606 0' | jc --proc             
    jc:  Error - Parser issue with proc:
                 ParseError: Proc file could not be identified.
                 ...

I can fix the "magic" signature (regex) to account for such cases.

4년 전discuss

It doesn't look like jc suffers from this bug since the regex match is greedy:

    % echo '2001 (my (file) with) S 1888 2001 1888 34816 2001 4202496 428 0 0 0 0 0 0 0 20 0 1 0 75513 115900416 297 18446744073709551615 4194304 5100612 140737020052256 140737020050904 140096699233308 0 65536 4 65538 18446744072034584486 0 0 17 0 0 0 0 0 0 7200240 7236240 35389440 140737020057179 140737020057223 140737020057223 140737020059606 0' | jc --proc

    {"pid":2001,"comm":"my (file) with","state":"S","ppid":1888,"pgrp":2001,"session":1888,"tty_nr":34816,"tpg_id":2001,"flags":4202496,"minflt":428,"cminflt":0,"majflt":0,"cmajflt":0,"utime":0,"stime":0,"cutime":0,"cstime":0,"priority":20,"nice":0,"num_threads":1,"itrealvalue":0,"starttime":75513,"vsize":115900416,"rss":297,"rsslim":18446744073709551615,"startcode":4194304,"endcode":5100612,"startstack":140737020052256,"kstkeep":140737020050904,"kstkeip":140096699233308,"signal":0,"blocked":65536,"sigignore":4,"sigcatch":65538,"wchan":18446744072034584486,"nswap":0,"cnswap":0,"exit_signal":17,"processor":0,"rt_priority":0,"policy":0,"delayacct_blkio_ticks":0,"guest_time":0,"cguest_time":0,"start_data":7200240,"end_data":7236240,"start_brk":35389440,"arg_start":140737020057179,"arg_end":140737020057223,"env_start":140737020057223,"env_end":140737020059606,"exit_code":0,"state_pretty":"Sleeping in an interruptible wait"}

Edit: looks like I can tighten up the signature matching regex for the "magic" syntax per the issue found above. The greedy regex matching for the parser does seem to work fine, though.

4년 전discuss

jc[0] supports proc files. Converts them to JSON or YAML. (I am the author)

[0] https://kellyjonbrazil.github.io/jc/docs/parsers/proc

4년 전discuss

Fortunately `jc`[0] does parse `/proc/<pid>/stat` correctly. I, of course, originally implemented it the naive/incorrect way until a contributor fixed it. :)

    $ cat /proc/2001/stat | jc --proc
    {"pid":2001,"comm":"my program with\nsp","state":"S","ppid":1888,"pgrp":2001,"session":1888,"tty_nr":34816,"tpg_id":2001,"flags":4202496,"minflt":428,"cminflt":0,"majflt":0,"cmajflt":0,"utime":0,"stime":0,"cutime":0,"cstime":0,"priority":20,"nice":0,"num_threads":1,"itrealvalue":0,"starttime":75513,"vsize":115900416,"rss":297,"rsslim":18446744073709551615,"startcode":4194304,"endcode":5100612,"startstack":140737020052256,"kstkeep":140737020050904,"kstkeip":140096699233308,"signal":0,"blocked":65536,"sigignore":4,"sigcatch":65538,"wchan":18446744072034584486,"nswap":0,"cnswap":0,"exit_signal":17,"processor":0,"rt_priority":0,"policy":0,"delayacct_blkio_ticks":0,"guest_time":0,"cguest_time":0,"start_data":7200240,"end_data":7236240,"start_brk":35389440,"arg_start":140737020057179,"arg_end":140737020057223,"env_start":140737020057223,"env_end":140737020059606,"exit_code":0,"state_pretty":"Sleeping in an interruptible wait"}

[0] https://kellyjonbrazil.github.io/jc/docs/parsers/proc_pid_st...

4년 전discuss

Both ways work:

    $ jc ifconfig lo0

    $ ifconfig lo0 | jc --ifconfig

4년 전discuss

Good news - `jc` supports JSONLines output for many commands and file-types, too!

4년 전discuss

Also note that you are looking at plaintext output here. By default `jc` and other JSON filtering tools do syntax highlighting when outputting to the terminal so it's actually quite easy to read JSON these days.

4년 전discuss

Hi - jc author here. Just a couple points:

The casting to int/float is not done unless the underlying values are predictably documented to be numbers. There are rare cases where auto int/float conversions are done, but:

1) This is always documented, and

2) You can turn this functionality off via the —raw flag

Also, predictable schemas are documented with each parser. (e.g `jc -h —-arp`)

4년 전discuss

Feel free to open issues on GitHub if you would like to recommend parsers! Also, the latest version of jc supports parsing of /proc files, which may also help.

4년 전discuss

Hi there! `jc` author here. Yes, it is a documented caveat[0] that the `C` or `en_US.UTF-8` locales should be used for best results.

It's not unheard of for tools to require `C` locale for proper parsing:

    $ LC_ALL=C ls -l | jc --ls

This is one of many inherent issues with using unstructured text as an API. That's why I believe there should be a JSON (or at least some other widely used format[1]) option for tools that have output that would be useful in scripts.

[0] https://github.com/kellyjonbrazil/jc#locale

[1] formats should have good library support across many languages and nice filter/query capabilities from the command-line

4년 전discuss

You can also run `jc -h --dig` to get the parser details that include the schema.

Having true JSON Schema[0] is being considered, but on the back-burner due to the sheer number of parsers to build schemas for. Also, it is more difficult to accurately define the schema for a small subset of parsers since their command output are so variable.

[0] https://json-schema.org/

4년 전discuss

`jc` is available today as a compiled binary with all dependencies self-contained. The binaries and OS package installers are available under the GitHub Releases:

https://github.com/kellyjonbrazil/jc/releases

This is still python under the hood and not as small of a binary as I would like, but it does work.

4년 전discuss

Hi there - author of `jc` here. I originally intended to have auto-detection but put that on the backburner to focus on creating parsers, especially after introducing the magic syntax.

I did implement auto-detection for `/proc` file parsers so you can just do:

    $ cat /proc/foo | jc --proc

or

    $ jc /proc/foo

But you can specify each procfile parser directly if you want to as well.

4년 전discuss

Hi there - `jc` author here. `jc` can also output in YAML format with the `-y` flag. It is fairly trivial to add other options in the future since `jc` just turns the text into objects which can be serialized to many different formats.

For example:

    % jc -y date
    ---
    year: 2022
    month: Nov
    month_num: 11
    day: 3
    weekday: Thu
    weekday_num: 4
    hour: 9
    hour_24: 9
    minute: 0
    second: 22
    period: AM
    timezone: PDT
    utc_offset:
    day_of_year: 307
    week_of_year: 44
    iso: '2022-11-03T09:00:22'
    epoch: 1667491222
    epoch_utc:
    timezone_aware: false

4년 전discuss

Hi there - I'm the author of `jc`. I also created `jello`[0], which works just like `jq` but uses python syntax. I find `jq` is great for many things but sometimes more complex operations are easier for me to grok in python.

[0] https://github.com/kellyjonbrazil/jello