CPU and GPU trends over time(r-bloggers.com)
r-bloggers.com
CPU and GPU trends over time
http://www.r-bloggers.com/cpu-and-gpu-trends-over-time/
26 comments
Very true. It is for the same reason that the article shows GPU progress outpacing CPU's in regards to FLOPS. The limited scope of a GPU's functionality allows most of the processor's silicon to be arithmetic units. CPU's, on the other hand, have huge sections of their real estate dedicated to handling conditional logic (something most GPU's fail miserably at), legacy instruction handling, virtualization, etc.
Thanks for posting this, its an excellent insight.
It was interesting when Intel fail to deliver on their roadmap. Here is a link from 2000 where Intel was alleged to be saying 10Ghz by 2011 http://www.geek.com/articles/chips/intel-predicts-10ghz-chip...
Which of course lead to the great Multi-core debate (AMD started it, Intel followed with two physical die on a package later to integrate into a single die).
Which lead to the great Memory channel bottleneck (Intel was in-order single channel, AMD was multi-channel) and resolution (everyone now is multi-channel).
GPUs on the other hand have always had memory bandwidth issues so they went wide, the GPU equivalent of multi-channel, from the start. They also benefit from a feature of graphics which was 'do this same algorithm on all of these texels' which is straight forward to parallelize.
And as nuclear weapons designers, and genomic researchers will tell you, GPUs are also useful in doing MIMD simulations.
Its less clear what the parallel architecture breakthrough is for OLAP type problems (or parallel file systems as an even easier example) These have lots of operations in parallel but they also have high connectivity of data synchronicity across those parallel threads which, at the moment, brings everything down to earth.
Fun times!
It was interesting when Intel fail to deliver on their roadmap. Here is a link from 2000 where Intel was alleged to be saying 10Ghz by 2011 http://www.geek.com/articles/chips/intel-predicts-10ghz-chip...
Which of course lead to the great Multi-core debate (AMD started it, Intel followed with two physical die on a package later to integrate into a single die).
Which lead to the great Memory channel bottleneck (Intel was in-order single channel, AMD was multi-channel) and resolution (everyone now is multi-channel).
GPUs on the other hand have always had memory bandwidth issues so they went wide, the GPU equivalent of multi-channel, from the start. They also benefit from a feature of graphics which was 'do this same algorithm on all of these texels' which is straight forward to parallelize.
And as nuclear weapons designers, and genomic researchers will tell you, GPUs are also useful in doing MIMD simulations.
Its less clear what the parallel architecture breakthrough is for OLAP type problems (or parallel file systems as an even easier example) These have lots of operations in parallel but they also have high connectivity of data synchronicity across those parallel threads which, at the moment, brings everything down to earth.
Fun times!
I would have preferred a logarithmic scale for the last graph (http://csgillespie.files.wordpress.com/2011/01/gpu3.png).
Here's one I just made using his data: http://imgur.com/TSFYB
Thanks. Looks more similar now than on the linear scale.
If someone figured out how to write regular apps on a GPU (which would amount to parallelizing the app), it could change everything.
1. What would be a first step towards this? It seems the big problem is that most code is sequential... and the opportunity is those parts that don't need to be, but just happen to be written that way.
2. Perhaps another opportunity is to try to remove the layers that make modern software so slow. Can they be distributed across all pipelines somehow?
3. Finally, a radical approach is not to try to convert existing apps, but to think about the whole problem of an app from the very beginning (what inputs will you get, what outputs do you want), and try to write that directly on a GPU. By making it concrete, you might see new tricks, perhaps by redefining the problem (the app's features).
1. What would be a first step towards this? It seems the big problem is that most code is sequential... and the opportunity is those parts that don't need to be, but just happen to be written that way.
2. Perhaps another opportunity is to try to remove the layers that make modern software so slow. Can they be distributed across all pipelines somehow?
3. Finally, a radical approach is not to try to convert existing apps, but to think about the whole problem of an app from the very beginning (what inputs will you get, what outputs do you want), and try to write that directly on a GPU. By making it concrete, you might see new tricks, perhaps by redefining the problem (the app's features).
This is what parallelizing Fortran compilers do. The space of programs that can be automatically parallelized is all computable programs (see http://en.wikipedia.org/wiki/Church–Rosser_theorem). The space of programs that can be efficiently automatically parallelized is very small.
Many problems aren't parallelizable for reasonable definitions of the word (i.e. polylogarithmic depth and polynomial work: Nick's class) unless there are some major breakthroughs in complexity theory.
As a class of problems, probably (although it hasn't been proven either way), but in terms of programs it comes down to the vagaries of data structures and algorithms - if the dataflow in your reduction graph makes it look like a linked list, there's only one way to reduce it.
Guy Steele gave a really good talk about this at ICFP 2009: http://labs.oracle.com/projects/plrg/Publications/ICFPAugust...
Guy Steele gave a really good talk about this at ICFP 2009: http://labs.oracle.com/projects/plrg/Publications/ICFPAugust...
The good thing is that functional programmers will barely notice that they are using the new data structures. (It's still just filters and maps. Just your fold-functions will take slightly different arguments.)
It can't really be done trivially. A GPU is a massively parallel (hundreds of cores), very limited type of CPU, specifically made to execute vector operations.
I agree, and I can't wait for someone (or, more likely, some group) to figure out how to parallelise arbitrary programs across GPUs and CPUs in a way that is transparent to the programmer - or as nearly so as possible.
I spend a lot of my "fun time" writing agent-based systems which involve multiple objects all running the same set of code within a single environment. A person could be forgiven for thinking that computers should be able to "just run it" on all these CPU or GPU cores but it doesn't work out that way right now unfortunately. I have tried Haskell but it's quite hard to do real work with, or maybe I am just dumb :-)
No doubt answers will come with time. Perhaps the "pain" of slow CPU cores will force one (or some) of us to put in some time to writing the type of solution I refer to in my first paragraph. (The related concept of "peak oil" springs to mind.)
I spend a lot of my "fun time" writing agent-based systems which involve multiple objects all running the same set of code within a single environment. A person could be forgiven for thinking that computers should be able to "just run it" on all these CPU or GPU cores but it doesn't work out that way right now unfortunately. I have tried Haskell but it's quite hard to do real work with, or maybe I am just dumb :-)
No doubt answers will come with time. Perhaps the "pain" of slow CPU cores will force one (or some) of us to put in some time to writing the type of solution I refer to in my first paragraph. (The related concept of "peak oil" springs to mind.)
If this chart from nVidia is to be believed, the CPU speed increase will continue for many years to come. They're projecting almost a 100x increase in speed from Tegra 2, in 2014!
http://images.anandtech.com/reviews/SoC/NVIDIA/Kal-El/roadma...
How are they going to get there? They could very easily pump up the shader count to get something that is "100x" performance, while not actually being 100x as powerful for general purpose operations. The graph looks shady to say the least.
Shit, I did not realize GPUs packed such a tremendous performance advantage over CPUs (FLOPs wise). They seem to be several orders of magnitude ahead--that is positively mindblowing!
And yet, all GPUs and CPUs of the whole world can just barely match one human brain (http://arstechnica.com/science/news/2011/02/adding-up-the-wo...). If that's not mindblowing, then what?
And yet, all GPUs and CPUs of the whole world can just barely match one human brain (http://arstechnica.com/science/news/2011/02/adding-up-the-wo...). If that's not mindblowing, then what?
They compare instructions per second with nerve impulses per second. That's a pretty useless comparison. I'm sure that all GPUs and CPUs could simulate a human brain more quickly than a human brain could simulate all CPUs and GPUs.
I'd venture to say that depends strongly on the problem the brain is trying to solve and the problem the processors are trying to solve.
In the Whole Brain Emulation Roadmap there's a table of published guesses at computing requirements: most were in the range 10^14..10^18 FLOPS. (Plus a few outliers from people who think the brain does molecular-scale or quantum computing in microtubules.) The 500th biggest computer in the latest Top500 does around 3E13 FLOPS, while the top does 3E15 -- using GPUs, incidentally. Your link's on the high end of the range, though they're all kind of guessing.
http://www.philosophy.ox.ac.uk/__data/assets/pdf_file/0019/3...
http://www.top500.org/list/2010/11/100
http://www.philosophy.ox.ac.uk/__data/assets/pdf_file/0019/3...
http://www.top500.org/list/2010/11/100
Looks like about 1 order of magnitude, not several.
What does "computational power" mean in this context? There's a graph but no explanation...
Probably number of floating point operations per second.
If you click on the graph, you can see "MIPS" (millions of instructions per second) on the dependent axis.
This seems like an apples to oranges comparison with different scales and units...
Data parallel algorithms with high computation-to-communication ratios, good data locality and low local branch divergence will see these kinds of speedups when rewritten for GPUs. This is an important subset of algorithms, but don't expect GPUs to replace CPUs for more general-purpose computation any time soon.