Command-line tools can be faster than your Hadoop cluster(aadrake.com)
aadrake.com
Command-line tools can be faster than your Hadoop cluster
http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
309 comments
Perhaps I'm missing something. It appears that the author is recommending against using Hadoop (and related tools) for processing 3.5GB of data. Who in the world thought that would be a good idea to begin with?
The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they can sound intelligent with a certain segment of the population.
That said, I like seeing how people put together their own CLI-based processing pipelines.
The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they can sound intelligent with a certain segment of the population.
That said, I like seeing how people put together their own CLI-based processing pipelines.
Hi all, original author here.
Some have questioned why I would spend the time advocating against the use of Hadoop for such small data processing tasks as that's clearly not when it should be used anyway. Sadly, Big Data (tm) frameworks are often recommended, required, or used more often than they should be. I know to many of us it seems crazy, but it's true. The worst I've seen was Hadoop used for a processing task of less than 1MB. Seriously.
Also, much agreement with those saying there should be more education effort when it comes to teaching command line tools. O'Reilly even has a book out on the topic: http://shop.oreilly.com/product/0636920032823.do
Thank you for all the comments and support.
Some have questioned why I would spend the time advocating against the use of Hadoop for such small data processing tasks as that's clearly not when it should be used anyway. Sadly, Big Data (tm) frameworks are often recommended, required, or used more often than they should be. I know to many of us it seems crazy, but it's true. The worst I've seen was Hadoop used for a processing task of less than 1MB. Seriously.
Also, much agreement with those saying there should be more education effort when it comes to teaching command line tools. O'Reilly even has a book out on the topic: http://shop.oreilly.com/product/0636920032823.do
Thank you for all the comments and support.
I think it is unsafe to parallelize grep with xargs as in done in the article, because, beyond delivery order shuffling, the output of the parallel greps could get mixed up (the beginning of a line is by one grep and the end of a line is from a different grep, so, reading line by line afterwards, you get garbled lines).
See https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B...
See https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B...
The example in the article with cat, grep and awk:
cat *.pgn | \
grep "Result" | \
awk '
{
split($0, a, "-");
res = substr(a[1], length(a[1]), 1);
if (res == 1) white++;
if (res == 0) black++;
if (res == 2) draw++;
}
END { print white+black+draw, white, black, draw }
'
awk '
/Result/ {
if (/1\/2/) draw++;
else if (/1-0/) white++;
else if (/0-1/) black++;
}
END { print white+black+draw, white, black, draw }
' *.pgnAuthor begins with fairly idiomatic shell pipeline, but in the search for performance the pipeline transforms to a awk script. Not that I have anything against awk, but I feel like that kinda runs against the premise of the article. The article ends up demonstrating the power of awk over pipelines of small utilities.
Another interesting note is that there is a possibility that the script as-is could mis-parse the data. The grep should use '^\[Result' instead of 'Result'. I think this demonstrates nicely the fragility of these sorts of ad-hoc parsers that are common in shell pipelines.
Another interesting note is that there is a possibility that the script as-is could mis-parse the data. The grep should use '^\[Result' instead of 'Result'. I think this demonstrates nicely the fragility of these sorts of ad-hoc parsers that are common in shell pipelines.
Bottom line is - you do not need hadoop until you cross 2TB of data to be processed (uncompressed).
Modern servers ( bare metal ones, not what AWS sells you ) are REALLY FAST and can crunch massive amounts of data.
Just use a proper tools, well optimized code written in C/C++/Go/etc - not all the crappy JAVA framework-in-a-framework^N architecture that abstracts thinking about the CPU speed.
Bottom line, the popular saying is true: "Hadoop is about writing crappy code and then running it on a massive scale."
Just use a proper tools, well optimized code written in C/C++/Go/etc - not all the crappy JAVA framework-in-a-framework^N architecture that abstracts thinking about the CPU speed.
Bottom line, the popular saying is true: "Hadoop is about writing crappy code and then running it on a massive scale."
Don't shoot me, but out of curiosity I wrote the thing in javascript: https://gist.github.com/ricardobeat/ee2fb2a6d704205446b7
Results: 4.4GB[1] processed in 47 seconds. Around 96mb/s, can probably be made faster, and nodejs is not the best at munging data...
[1] 3201 files taken from http://github.com/rozim/ChessData
Results: 4.4GB[1] processed in 47 seconds. Around 96mb/s, can probably be made faster, and nodejs is not the best at munging data...
[1] 3201 files taken from http://github.com/rozim/ChessData
This article echoes a talk Bryan Cantrill gave two years ago:
https://youtu.be/S0mviKhVmBI
It's about how Joyent took the concept of a UNIX pipeline as a true powertool and built a distributed version atop an object filesystem with some little map/reduce syntactic sugar to replace Hadoop jobs with pipelines.
The Bryan Cantrill talk is definitely worth your time, but you can get an understanding of Manta with their 3m screencast: https://youtu.be/d2KQ2SQLQgg
It's about how Joyent took the concept of a UNIX pipeline as a true powertool and built a distributed version atop an object filesystem with some little map/reduce syntactic sugar to replace Hadoop jobs with pipelines.
The Bryan Cantrill talk is definitely worth your time, but you can get an understanding of Manta with their 3m screencast: https://youtu.be/d2KQ2SQLQgg
Next to using `xargs -P 8 -n 1` to parallellize jobs locally, take a look at paexec, GNU parallel replacement that just works.
See https://github.com/cheusov/paexec
See https://github.com/cheusov/paexec
See this very good comment by Bane:
https://news.ycombinator.com/item?id=8902739
https://news.ycombinator.com/item?id=8902739
I had an intern over the summer, working on a basic A/B Testing framework for our application (a very simple industrial handscanner tool used inside warehouses by a few thousand employees).
When we came to the last stage, analysis, he was keen to use MapReduce so we let him. In the end though, his analysis didn't work well, took ages to process when it did, and didn't provide the answers we needed. The code wasn't maintainable or reusable. shrug It happens. I had worse internships.
I put together some command line scripts to parse the files instead- grep, awk, sed, really basic stuff piped into each other and written to other files. They took 10 minutes or so to process, and provided reliable answers. The scripts were added as an appendix to the report I provided on the A/B test, and after formatting and explanations, took up a couple pages.
When we came to the last stage, analysis, he was keen to use MapReduce so we let him. In the end though, his analysis didn't work well, took ages to process when it did, and didn't provide the answers we needed. The code wasn't maintainable or reusable. shrug It happens. I had worse internships.
I put together some command line scripts to parse the files instead- grep, awk, sed, really basic stuff piped into each other and written to other files. They took 10 minutes or so to process, and provided reliable answers. The scripts were added as an appendix to the report I provided on the A/B test, and after formatting and explanations, took up a couple pages.
We have a proprietary algorithm for assigning foods a "suitability score" based on a user's personal health conditions and body data.
It used to be a fairly slow algorithm, so we ran it in a hadoop cluster and it cached the scores for every user vs. every food in a massive table on a distributed database.
Another developer, who is quite clever, rewrote our algorithm in C, and compiled it as a database function, which was about 100x faster. He also did some algebra work and found a way to change our calculations, yielding a measly 4-5x improvement.
It was so, so, so much faster that in one swoop we eliminated our entire Hadoop cluster, and the massive scores table, and were actually able sort your food search results by score, calculating scores on the fly.
It used to be a fairly slow algorithm, so we ran it in a hadoop cluster and it cached the scores for every user vs. every food in a massive table on a distributed database.
Another developer, who is quite clever, rewrote our algorithm in C, and compiled it as a database function, which was about 100x faster. He also did some algebra work and found a way to change our calculations, yielding a measly 4-5x improvement.
It was so, so, so much faster that in one swoop we eliminated our entire Hadoop cluster, and the massive scores table, and were actually able sort your food search results by score, calculating scores on the fly.
This also isn't a straight either or proposition. I build local command line pipelines and do testing and/or processing. When either the amount of data needed to be processed passes into the range where memory or network bandwidth makes the processing more efficient on a Hadoop cluster I make some fairly minimal conversions and run the stream processing on the Hadoop cluster in streaming mode. It hasn't been uncommon for my jobs to be much faster than the same jobs run on the cluster with Hive or some other framework. Much of the speed boils down to the optimizer and the planner.
Overall I find it very efficient to use the same toolset locally and then scale it up to a cluster when and if I need to.
Overall I find it very efficient to use the same toolset locally and then scale it up to a cluster when and if I need to.
If bash is the shell (assuming recursive search is required), maybe it would be even faster to just do:
shopt -s globstar
mawk '/Result/ {
game++
split($0, a, "-")
res = substr(a[1], length(a[1]), 1)
if(res == 1)
white++
if(res == 0)
black++
if(res == 2)
draw++
} END {
print game, white, black, draw
}' **/*.pgn
?This is a great exercise of how to take a Unix command line and iteratively optimize it with advanced use of awk.
In that spirit, one can optimize the xargs mawk invocation by 1) Getting rid of string-manipulation function calls (which are slow in awk), 2) using regular expressions in the pattern expression (which allows awk to short-circuit the evaluation of lines), and 3) avoiding use of field variables like $1, and $2, which allows the mawk virtual machine to avoid implicit field splitting. A bonus is that you end up with an awk script which is more idiomatic:
In that spirit, one can optimize the xargs mawk invocation by 1) Getting rid of string-manipulation function calls (which are slow in awk), 2) using regular expressions in the pattern expression (which allows awk to short-circuit the evaluation of lines), and 3) avoiding use of field variables like $1, and $2, which allows the mawk virtual machine to avoid implicit field splitting. A bonus is that you end up with an awk script which is more idiomatic:
mawk '
/^\[Result "1\/2-1\/2"\]/ { draw++ }
/^\[Result "1-0"\]/ { white++ }
/^\[Result "0-1"\]/ { black++ }
END { print white, black, draw }'
mawk '
{games += ($1+$2+$3); white += $1; black += $2; draw += $3}
END { print games, white, black, draw }'
First, you don't score points with me for saying not to use Hadoop when you don't need to use Hadoop.
Second, you don't get to pretend you invented shell scripting because you came up with a new name for it.
Third, there are very few cases if any where writing a shell script is better than writing a Perl script.
Second, you don't get to pretend you invented shell scripting because you came up with a new name for it.
Third, there are very few cases if any where writing a shell script is better than writing a Perl script.
To quote the memorable Ted Dziuba[0]:
"Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ.
The Taco Bell answer? xargs and wget. In the rare case that you saturate the network connection, add some split and rsync. A "distributed crawler" is really only like 10 lines of shell script."
[0] since his blog is gone: http://readwrite.com/2011/01/22/data-mining-and-taco-bell-pr...
"Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ.
The Taco Bell answer? xargs and wget. In the rare case that you saturate the network connection, add some split and rsync. A "distributed crawler" is really only like 10 lines of shell script."
[0] since his blog is gone: http://readwrite.com/2011/01/22/data-mining-and-taco-bell-pr...
I feel ag (silver surfer, a grep-ish alternative) should be mentioned (even though he dropped it in his final awk/mawk commands) as it tends to be much faster than grep, and considering he cites performance throughout.
A similar story: http://blogs.law.harvard.edu/philg/2009/05/18/ruby-on-rails-...: Tools used not quite the right way.
edit: with HN commentary: https://news.ycombinator.com/item?id=615587
edit: with HN commentary: https://news.ycombinator.com/item?id=615587
on a couple of GB this is true, actually if you have ssd's I'd expect any non compute bound task to be faster on a single machine up to ~10gb after which the disk parallelism should kick in and Hadoop should start to win.
So don't use Hadoop to crunch data that fits on a memory stick, or that a single disk spindle can read in few seconds.
Why is this first on the HN front-page?
Reminds me of the C++ is better than Java, Go is better than C++, etc, pieces.
Yes, the right tool for the right job. That's what makes a good engineer.
Somebody who thinks there is _no_ valid use case for Hadoop is a fool. (The author did not say that, but many of the comments here seem to imply that view)
Why is this first on the HN front-page?
Reminds me of the C++ is better than Java, Go is better than C++, etc, pieces.
Yes, the right tool for the right job. That's what makes a good engineer.
Somebody who thinks there is _no_ valid use case for Hadoop is a fool. (The author did not say that, but many of the comments here seem to imply that view)
Here is an analysis from a developer who looked at Hadoop-
http://ossectools.blogspot.ca/2012/03/why-elsa-doesnt-use-ha...
(ELSA is a logger that claims to be able to handle 100000 entries/sec (!!))
When to Use Hadoop
This is a description of why Hadoop isn't always the right solution to Big Data problems, but that certainly doesn't mean that it's not a valuable project or that it isn't the best solution for a lot challenges. It's important to use the right tool for the job, and thinking critically about what features each tool provides is paramount to a project's success. In general, you should use Hadoop when:
(ELSA is a logger that claims to be able to handle 100000 entries/sec (!!))
When to Use Hadoop
This is a description of why Hadoop isn't always the right solution to Big Data problems, but that certainly doesn't mean that it's not a valuable project or that it isn't the best solution for a lot challenges. It's important to use the right tool for the job, and thinking critically about what features each tool provides is paramount to a project's success. In general, you should use Hadoop when:
Data access patterns will be very basic but analytics will be very complicated.
Your data needs absolutely guaranteed availability for both reading and writing.
There are inadequate traditional database-oriented tools which currently exist for your problem.
You're don't know exactly why you're using it.
You want to maximize hardware efficiency.
Your data fits on a single "beefy" server.
You don't have full-time staff to dedicate to it.
Shell commands are great for data processing pipelines because you get parallelism for free. For proof, try a simple example in your terminal.
Also, the parallelism of a data processing pipeline is going to be constrained by the speed of the slowest process in it: all the processes after it are going to be idle while waiting for the slow process to produce output, and all the processes before it are going to be idle once the slow process has filled its pipe's input buffers. So if one of the processes in the pipeline takes 100 times as long as the other three, Amdahl's Law[1] suggests that you won't get a big win from breaking it up into multiple processes.
[1] https://en.wikipedia.org/wiki/Amdahl%27s_law
Edit: As someone pointed out, my example needed "grep -h". Fixed.
sleep 3 | echo "Hello world."
cat *.pgn | grep "Result" | sort | uniq -c
grep -h "Result" *.pgn | sort | uniq -c
Also, the parallelism of a data processing pipeline is going to be constrained by the speed of the slowest process in it: all the processes after it are going to be idle while waiting for the slow process to produce output, and all the processes before it are going to be idle once the slow process has filled its pipe's input buffers. So if one of the processes in the pipeline takes 100 times as long as the other three, Amdahl's Law[1] suggests that you won't get a big win from breaking it up into multiple processes.
[1] https://en.wikipedia.org/wiki/Amdahl%27s_law
Edit: As someone pointed out, my example needed "grep -h". Fixed.
About 5 years ago I worked at a company that took the "pile of shell scripts" approach to processing data. Our data was big enough and our algorithms computationally heavy enough that a single machine wasn't a good solution. So we had a bunch of little binaries that were glued together with sed, awk, perl, and pbsnodes.
It was horrible. It was tough to maintain-- we all know how hard to read even the best awk and perl are. It was difficult to optimize, and you always found yourself worrying about things like the maximum length of command lines, how to figure out what the "real" error was in a bash pipeline, and so on. When parts of the job failed, we had to manually figure out what parts of the job had failed, and re-run them. Then we had to copy the files over to the right place to create the full final output.
The company was a startup and the next VC milestone or pivot was always just around the corner. There was never any time to clean things up. A lot of the code had come out of early tech demos that management just asked us to "just scale up." But oops, you can't do that with a pile of shell scripts and custom C binaries. So the technical debt just kept piling up. I would advise anyone in this situation not to do this. Yeah, shell scripts are great for making rough guesses about things in a pile of data. They are great for ad hoc exploration on small data or on individual log files. But that's it. Do not check them into a source code repo and don't use them in production. The moment someone tries to check in a shell script longer than a page, you need to drop the hammer. Ask them to rewrite it in a language (and ideally, framework), that is maintainable in the long term.
Now I work on Hadoop, mostly on the storage side of things. Hadoop is many things-- a storage system, a set of computation frameworks that are robust against node failures, a Java API. But above all it's a framework for doing things in a standardized way so that you can understand what you've done 6 months from now. And you will be able to scale up by adding more nodes, when your data is 2x or 4x as big down the line. On average, the customers we work with are seeing their data grow by 2x every year.
I feel like people on Hacker News often don't have a clear picture of how people interact with Hadoop. Writing MapReduce jobs is very 2008. Nowadays, more than half of our users write SQL that gets processed by an execution engine such as Hive or Impala. Most users are not developers, they're analysts. If you have needs that go beyond SQL, you would use something like Spark, which has a great and very concise API based on functional programming. Reading about how clunky MR jobs is just feels to me like reading an article about how hard it is to make boot and root floppy disks for Linux. Nobody's done that in years.
It was horrible. It was tough to maintain-- we all know how hard to read even the best awk and perl are. It was difficult to optimize, and you always found yourself worrying about things like the maximum length of command lines, how to figure out what the "real" error was in a bash pipeline, and so on. When parts of the job failed, we had to manually figure out what parts of the job had failed, and re-run them. Then we had to copy the files over to the right place to create the full final output.
The company was a startup and the next VC milestone or pivot was always just around the corner. There was never any time to clean things up. A lot of the code had come out of early tech demos that management just asked us to "just scale up." But oops, you can't do that with a pile of shell scripts and custom C binaries. So the technical debt just kept piling up. I would advise anyone in this situation not to do this. Yeah, shell scripts are great for making rough guesses about things in a pile of data. They are great for ad hoc exploration on small data or on individual log files. But that's it. Do not check them into a source code repo and don't use them in production. The moment someone tries to check in a shell script longer than a page, you need to drop the hammer. Ask them to rewrite it in a language (and ideally, framework), that is maintainable in the long term.
Now I work on Hadoop, mostly on the storage side of things. Hadoop is many things-- a storage system, a set of computation frameworks that are robust against node failures, a Java API. But above all it's a framework for doing things in a standardized way so that you can understand what you've done 6 months from now. And you will be able to scale up by adding more nodes, when your data is 2x or 4x as big down the line. On average, the customers we work with are seeing their data grow by 2x every year.
I feel like people on Hacker News often don't have a clear picture of how people interact with Hadoop. Writing MapReduce jobs is very 2008. Nowadays, more than half of our users write SQL that gets processed by an execution engine such as Hive or Impala. Most users are not developers, they're analysts. If you have needs that go beyond SQL, you would use something like Spark, which has a great and very concise API based on functional programming. Reading about how clunky MR jobs is just feels to me like reading an article about how hard it is to make boot and root floppy disks for Linux. Nobody's done that in years.
I've had the pleasure and displeasure of working with small datasets (~7.5GB of images) in shell. One often needs to send SIGINT to the shell when it starts to glob expand or tab complete a folder with millions of files. But besides minor issues like that, command line tools get the job done.
'xargs -n' elicits fond memories of spawning large jobs to my Openmosix cluster ! I miss Openmosix.
There's a perception that hadoop commands are terribly complex. If you run
$spark-shell
you can execute (interactively)
val file = spark.textFile("hdfs://...") val errors = file.filter(line => line.contains("ERROR")) errors.count()
And wordcount a file - ok the wget is not there, but this is really not complex!
$spark-shell
you can execute (interactively)
val file = spark.textFile("hdfs://...") val errors = file.filter(line => line.contains("ERROR")) errors.count()
And wordcount a file - ok the wget is not there, but this is really not complex!
This kind of approach can probably scale out pretty far before actually needing to resort to true distributed processing. Compression, simple transforms, R, etc... You can probably get away with even more by just using a networked filesystem and inotify.
One common misconception about using Hadoop is that use Hadoop if your data is large. Usage of Hadoop should be more driven based on the growth of data rather than size.
I agree that for the given use case, the solution is appropriate and works fine. Problem mentioned in the given post is not a Big Data problem.
Hadoop will be helpful in case if there are millions of games are played everyday and we need to update the statistics daily e.t.c. For this case, the given solution will hit bottleneck and there will be some optimisation/code change needed to keep running the code.
Hadoop and its ecosystem are not a silver bullet and hence should not be used for everything. The problem has to be a Big Data problem
I agree that for the given use case, the solution is appropriate and works fine. Problem mentioned in the given post is not a Big Data problem.
Hadoop will be helpful in case if there are millions of games are played everyday and we need to update the statistics daily e.t.c. For this case, the given solution will hit bottleneck and there will be some optimisation/code change needed to keep running the code.
Hadoop and its ecosystem are not a silver bullet and hence should not be used for everything. The problem has to be a Big Data problem
That it also happens to very fast and powerful (when memory isn't a limiting factor) is nice icing on the cake. I moved over to doing much more on CLI after realizing that doing something as simple as "head -n 1 massive.csv" to inspect headers of corrupt multi-gb CSV files made my data-munging life substantially more enjoyable than opening them up in Sublime Text.