Jq - a lightweight and flexible command-line JSON processor(stedolan.github.com)
stedolan.github.com
Jq - a lightweight and flexible command-line JSON processor
http://stedolan.github.com/jq/
19 comments
Arch PKGBUILD would be much easier with a version number too...
They exist now!
Very nice tool! I'm getting syntax errors from the Twitter examples though (and everytime I try to use underscores):
<edit>Created an issue on Github</edit>
<edit>Created an issue on Github</edit>
curl 'http://search.twitter.com/search.json?q=json&rpp=5&include_entities=true | jq '.results[] | {from_user, text, urls: [.entities.urls[].url]}'
error: Invalid character
.results[] | {from_user, text, urls: [.entities.urls[].url]}
^
error: syntax error, unexpected IDENT, expecting '}'
.results[] | {from_user, text, urls: [.entities.urls[].url]}
^^^^
2 compile errors
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5970 100 5970 0 0 11969 0 --:--:-- --:--:-- --:--:-- 23050(author here) That bug was fixed a while ago, but I forgot to upload new binaries. Try again, they should be working now.
Post it here...
https://github.com/stedolan/jq/issues
https://github.com/stedolan/jq/issues
Same issue here.
See also, underscore-cli https://github.com/ddopson/underscore-cli
Also see also RecordStream: https://github.com/benbernard/RecordStream
Someone mentioned jsawk in a sibling comment, too.
Someone mentioned jsawk in a sibling comment, too.
Also see http://www.kmkeen.com/jshon/
> You can download a single binary, scp it to a far away machine, and expect it to work.
Not if my machine is x86 and the server is SPARC.
Cool utility though.
Not if my machine is x86 and the server is SPARC.
Cool utility though.
Git clone and make, then. As long as you have a C compiler. And if you don’t, you probably aren’t using JSON.
Actually I bet there are a ton of people using JSON that don't have a c compiler. The technologies are spaces by several decades and I would imagine the majority of JSON people are using much higher level languages than c.
If I want to munge JSON I find it easier to do so in python, just convert it back to data structures and manipulate them and serialized it back to JSON.
If I want to munge JSON I find it easier to do so in python, just convert it back to data structures and manipulate them and serialized it back to JSON.
I would usually use some other language with a JSON library as well, but this utility is conveniently one-function, after Unix fashion. I like composability. Also, a C compiler is usually on hand whether you use it or not; even if none is, they’re not exactly hard to obtain.
"they're not exactly hard to obtain"
Except, of course, in Mac OS X where you have to download (a couple of gigabytes) Xcode from Apple and install it to get GCC :)
Except, of course, in Mac OS X where you have to download (a couple of gigabytes) Xcode from Apple and install it to get GCC :)
<The technologies are spaces by several decades
this intrigues me. while the widespread use of JSON as JSON is perhaps what differentiates it from what i'm about to describe i propose that the concept is an inevitable or obvious means of data representation/manipulation. As an arguably irrelevant anecdote to support this proposition I submit that prior to my personal introduction to the existence of JSON I was organizing data in strings and in txt files in a format nearly identical to basic JSON and writing simple functions to essentially do the same thing. I submit that I am not special or particularly clever and therefore the baller-coders that cometh before me certainly had to have their own types of crap like this. If i didn't have a tablet in class i probably would have had their names in my notes, but i digress.
tl;dr - While the advent of JSON's incarnation and that of C are several decades apart the underlying concepts are not.
tl;dr;dr - just realized your second sentence kinda says this.
this intrigues me. while the widespread use of JSON as JSON is perhaps what differentiates it from what i'm about to describe i propose that the concept is an inevitable or obvious means of data representation/manipulation. As an arguably irrelevant anecdote to support this proposition I submit that prior to my personal introduction to the existence of JSON I was organizing data in strings and in txt files in a format nearly identical to basic JSON and writing simple functions to essentially do the same thing. I submit that I am not special or particularly clever and therefore the baller-coders that cometh before me certainly had to have their own types of crap like this. If i didn't have a tablet in class i probably would have had their names in my notes, but i digress.
tl;dr - While the advent of JSON's incarnation and that of C are several decades apart the underlying concepts are not.
tl;dr;dr - just realized your second sentence kinda says this.
Also see App::PipeFilter - https://metacpan.org/module/App::PipeFilter
The module is a framework/collection of shell pipelines which include JSON processing:
The module is a framework/collection of shell pipelines which include JSON processing:
curl -s 'http://api.duckduckgo.com/?q=poe&o=json' |
jsonpath -o '$..Topics.*.FirstURL' -o '$..Topics.*.Text' |
grep -i perl |
jmap -i col0 -o url -i col1 -o title |
json2yaml
title: Perl Object Environment, a library for event driven multitasking for the Perl programming language
url: http://duckduckgo.com/Perl_Object_Environment
Wow, this compiles fast. And the Flex and Bison files are there. This seems hackable. Nice work!
My only questions are (they are always the same with any purported sed/awk "replacement"):
1. what was the problem you were trying to solve where sed and awk failed you, and
2. does this program operate line by line or does it read entire files into memory?
I had to deal with some JSON a while ago and threw together some sed like this just so I could read it:
My only questions are (they are always the same with any purported sed/awk "replacement"):
1. what was the problem you were trying to solve where sed and awk failed you, and
2. does this program operate line by line or does it read entire files into memory?
I had to deal with some JSON a while ago and threw together some sed like this just so I could read it:
sed '
s/,/&\
/g;
/^$/d;
s/^[{][^}]/\
&/g;
/\"/s/,/<##eol##>/g;
s/ *//;
' |tr '\012' '\040' |sed '
s/<##eol##>/\
/g;
s/\[/\
&\
/;
s/\]/\
&\
/;
s/, /, /g;
'
Thanks!
1. I think your example shows sed/awk's failings with JSON data :) I don't want to write a JSON parser by hand every time I want to pull a field out of an object, and parsing recursive structures with regexes is never a good plan.
2. It reads JSON items from stdin into memory, one at a time. So if the input is a single giant JSON value, it all goes into memory, but if it's a series of whitespace-separated values they'll be processed sequentially.
It's cat-friendly: if you do
1. I think your example shows sed/awk's failings with JSON data :) I don't want to write a JSON parser by hand every time I want to pull a field out of an object, and parsing recursive structures with regexes is never a good plan.
2. It reads JSON items from stdin into memory, one at a time. So if the input is a single giant JSON value, it all goes into memory, but if it's a series of whitespace-separated values they'll be processed sequentially.
It's cat-friendly: if you do
cat a | jq foo; cat b | jq foo
cat a b | jq foo1. But those are general statements. Opinions. What I mean is give me a specific case. A specific example, a specific block of JSON and a specific task. Once I have that, then I can ask myself "Is this something I would ever need to do or that I have to do on a regular basis?"
Sometimes I need to write one-off filters. There is just no getting around it. I have to choose a utility that gives maximum flexibility and is not too verbose; I don't like to type. Lots of people like Perl, and other similar scripting languages for writing one-off filters. But Perl, _out of the box_, is not a line-by-line filter. It's unlike sed/awk; it needs more memory. That brings us to #2.
2. If I understand correctly, jq is reading the entire JSON block into memory. This is what separates your program and so many other sed/awk "replacements" from sed and awk, the programs they purport to "replace". sed/awk don't read entire files into memory, they operate line-by-line and use a reasonably small buffer. Any sed/awk "replacement" would have to match that functionality. Given that sed/awk don't read in an entire structure (JSON, XML, etc.) before processing it, they are ideal for memory constrained environments. (As long as you don't overfill their buffers, which rarely happens in my experience.)
Anyway, so far I like this program. Best JSON filter I've seen yet (because I can hack on the lexer and parser you provided).
Well done.
Sometimes I need to write one-off filters. There is just no getting around it. I have to choose a utility that gives maximum flexibility and is not too verbose; I don't like to type. Lots of people like Perl, and other similar scripting languages for writing one-off filters. But Perl, _out of the box_, is not a line-by-line filter. It's unlike sed/awk; it needs more memory. That brings us to #2.
2. If I understand correctly, jq is reading the entire JSON block into memory. This is what separates your program and so many other sed/awk "replacements" from sed and awk, the programs they purport to "replace". sed/awk don't read entire files into memory, they operate line-by-line and use a reasonably small buffer. Any sed/awk "replacement" would have to match that functionality. Given that sed/awk don't read in an entire structure (JSON, XML, etc.) before processing it, they are ideal for memory constrained environments. (As long as you don't overfill their buffers, which rarely happens in my experience.)
Anyway, so far I like this program. Best JSON filter I've seen yet (because I can hack on the lexer and parser you provided).
Well done.
Got the following compile error when running make (Lion + XCode 4.5.1):
bison -W -d parser.y -v --report-file=parser.gen.info -o
parser.gen.c
bison: invalid option -- W
Try `bison --help' for more information.
make: *** [parser.gen.c] Error 1(author here) I haven't tried to build on a mac in a while, bison there seems to support fewer options (must be an older version).
For now, I've just checked in the autogenerated parser, so that bison won't have to run when you build master. git-pull and try again :)
For now, I've just checked in the autogenerated parser, so that bison won't have to run when you build master. git-pull and try again :)
Works great now, thank you:)
FYI if you have python installed (and most do) you can just do
curl 'http://search.twitter.com/search.json?q=json&rpp=5&i... | python -mjson.tool
curl 'http://search.twitter.com/search.json?q=json&rpp=5&i... | python -mjson.tool
Very nice tool, congrats!
This is something everyone ends up needing at least one time, and we always end up using a language specific tool.
Nothing beats the command line piping
This is something everyone ends up needing at least one time, and we always end up using a language specific tool.
Nothing beats the command line piping
[deleted]
See also, jsawk: https://github.com/micha/jsawk
[deleted]
When doing the tasks this seems to be targeting, Jansson has worked well for me.
http://www.digip.org/jansson/
http://www.digip.org/jansson/
Built for Windows using cygwin, works a treat.
Builds with MinGW as well, although I did have to manually edit a header file because I couldn't get sed to work properly.
See also jsontool, http://trentm.com/json/
We periodically have to munge JSON at work—will definitely be trying this out.
This is awesome. Thank you.
Thanks; very nicely done. :-)
D. DQ,-"l
Also, if you tag releases with a version number and make them available as downloads on GitHub, you'll have an easier time getting Jq into Homebrew, the OS X package manager. I wrote a formula for you, but AFAIK you'll need to add in a real version number before submitting a pull request to Homebrew: https://gist.github.com/3928074