Getting Unique Counts From a Log File(blog.hypergeometric.com)

9 pointsby gpapilionقبل 13 سنة8 comments

blog.hypergeometric.com

http://blog.hypergeometric.com/2013/06/22/getting-unique-counts-from-a-log-file/

3 comments

It amazes me at work how many people just violate UUOC/UUOG

ie..

  cat file | grep bar | awk '{ print $1 }' | sort | uniq

which can simply be

  awk '/bar/ { print $1 | "sort -u"}' file

cascaقبل 13 سنة

I've done this for single-use pipelines when built up because the nature of the data is unfamiliar. It usually starts with

   cat file | less

and then

   cat file | grep bar | less

and then

   cat file | grep bar | awk '{ print $1 }' | less

and so on.

But if it's going to be used more than once then it does seem odd to have the unnecessary commands. Maybe "advanced" awk (beyond print $1) is unknown to people?

ibottyقبل 13 سنة

if you are only doing 'print $1' you should be using cut. but whatever.

gpapilionقبل 13 سنة

Its a balance of clarity, vs efficiency. Also, I chose uniq over sort -u, for the -c option.

That being said, point well made.

e12eقبل 13 سنة

Well, I often do:

    cat some.log | awk '/regexp/ { print $2}' | sort| uniq

Then tweak that line to get what I want, then change it to:

    zcat some.log*| ...

jonjenkقبل 13 سنة

There is a great wiki book on this subject.

http://en.wikibooks.org/wiki/Ad_Hoc_Data_Analysis_From_The_U...

pessimizerقبل 13 سنة

Hopefully you've already submitted this, because I'm about to. Looks great.

steffanقبل 13 سنة

I believe the word you want is 'Glean'