Benchmarking JS(mrale.ph)
mrale.ph
Benchmarking JS
http://mrale.ph/talks/goto2015/#/
33 comments
Thanks, I am glad you like my blog :)
I might write something about TF but right now it's still not used to compile normal code and the team is just starting on stuff like adaptive optimizations, so it's a bit too early.
Also given that I don't work on V8 anymore I always think it's a bit unfair for me to do it --- somebody from V8 team should do it instead, cause my perspective is bound to be slightly different from theirs :)
I might write something about TF but right now it's still not used to compile normal code and the team is just starting on stuff like adaptive optimizations, so it's a bit too early.
Also given that I don't work on V8 anymore I always think it's a bit unfair for me to do it --- somebody from V8 team should do it instead, cause my perspective is bound to be slightly different from theirs :)
And, since you're reading this (in the other comments) I just want to point out that the presentations you're giving got a lot better (the deck). This one is a lot more witty than the old ones. Props on getting better at presenting - a big step from your old presentations and blog posts, the English is good too here :)
Agree : ) Deff. awesome experience watching him present. Such talks stick with you through out.
Does anybody happen to have the video for the talk? I'd be much obliged. Just the slides leave a bunch of holes I want to fill in.
I can only find the videos from 2014.
I can only find the videos from 2014.
From https://www.youtube.com/playlist?list=PLEx5khR4g7PJPfVDB1nUp... it looks like you might be waiting a while - 2014 conference was in May, videos went up early/mid August.
Adding more info: https://twitter.com/GOTOchgo/status/598489575481286656
Sign up to their newsletter at http://gotocon.com/chicago-2015/
Sign up to their newsletter at http://gotocon.com/chicago-2015/
It was recorded but I don't know when it will be publicly released.
This talk is a combination of two separate talks I have given before plus two new examples, based on my recent endeavors.
Talks merged here are LXJS 2013 one[1] where I talk about microbenchmarking pitfalls and WebRebels 2014 one[2] where describe how bugs in VMs can affect benchmarking results.
If you have any questions about the slides just ask them here or send me an email --- I will try to answer as soon as I can.
[1] https://www.youtube.com/watch?v=65-RbBwZQdU
[2] https://webrebels.23video.com/crooked-mirrors-of-performance...
This talk is a combination of two separate talks I have given before plus two new examples, based on my recent endeavors.
Talks merged here are LXJS 2013 one[1] where I talk about microbenchmarking pitfalls and WebRebels 2014 one[2] where describe how bugs in VMs can affect benchmarking results.
If you have any questions about the slides just ask them here or send me an email --- I will try to answer as soon as I can.
[1] https://www.youtube.com/watch?v=65-RbBwZQdU
[2] https://webrebels.23video.com/crooked-mirrors-of-performance...
Your talks always frighten me when it comes to VM's. I just want predictable performance- is that too much to ask? (Semi-serious question).
edit: to clarify. In C++ I write dumb but readable code knowing that the compiler will always perform specific optimizations. In JS it seems that there are many factors resulting in different optimization levels. How do I ensure that the VM does not execute my 'dumb but readable' code literally? Without that guarantee it seems that I have to frequently resort to optimized yet unreadable code.
edit: to clarify. In C++ I write dumb but readable code knowing that the compiler will always perform specific optimizations. In JS it seems that there are many factors resulting in different optimization levels. How do I ensure that the VM does not execute my 'dumb but readable' code literally? Without that guarantee it seems that I have to frequently resort to optimized yet unreadable code.
I obviously craft/collect examples of V8 going off the rails for these talks just to show that VMs are software and all software has bugs --- and those bugs don't necessarily manifest as crashes and incorrect results - they can lead to worse performance and developers must be ready for this: must be ready to diagnose these issues, report them and work around them.
For a single test case where V8 goes off the rails there are millions of lines of code across the globe which V8 optimizes correctly.
On a funny note: I actually do have a version of this talk where I show GCC going slightly of the rails and producing a code that is 3 times slower than it should be because it hits an (infamous) partial register dependency stall --- see StackOverflow question[1] for the gory details.
> How do I ensure that the VM does not execute my 'dumb but readable' code literally?
Well, as I do say in the talk: reasonable code should be reasonably fast. If it is not the case --- file bugs with VM vendors.
Keeping your code relatively static / monomorphic is the best way to achive performance in any language.
In any case I think it's much much much more important to optimize algorithms not their concrete implementations.
[1] http://stackoverflow.com/questions/26585977/64-bit-code-gene...
For a single test case where V8 goes off the rails there are millions of lines of code across the globe which V8 optimizes correctly.
On a funny note: I actually do have a version of this talk where I show GCC going slightly of the rails and producing a code that is 3 times slower than it should be because it hits an (infamous) partial register dependency stall --- see StackOverflow question[1] for the gory details.
> How do I ensure that the VM does not execute my 'dumb but readable' code literally?
Well, as I do say in the talk: reasonable code should be reasonably fast. If it is not the case --- file bugs with VM vendors.
Keeping your code relatively static / monomorphic is the best way to achive performance in any language.
In any case I think it's much much much more important to optimize algorithms not their concrete implementations.
[1] http://stackoverflow.com/questions/26585977/64-bit-code-gene...
C++ performance is a true minefield between different compilers and their versions. For example, fail to provide correct incantation of compiler flags or do something in code that confuses auto-vectorization, and your performance can suffer by an order of magnitude.
I guess a lot of Javascript micro-optimization comes down to don't change field type and give type hints in assignments. Like:
Emscripten can be used to compile C++ into Javascript. Perhaps it could help you?
I guess a lot of Javascript micro-optimization comes down to don't change field type and give type hints in assignments. Like:
var integerType = 0|0;
integerType = 45|0;
integerType = (integerType + 1)|0;
Emscripten can be used to compile C++ into Javascript. Perhaps it could help you?
Really enjoyed that LXJS 2013 video the first time I saw it highlighted on Dave Methvin's blog ("Please Stop the jsPerf.com Abuse!"). Hilarious and very informative - thanks!
Okay, thank you! I enjoyed the slides and those talks will give me a lot of study material for tonight.
Thanks again!
Thanks again!
Thanks!
I found this very interesting since I'm working on something similar where I need to be able to measure how long certain operations take.
And I had no clue that js-engines has come this far, optimizing code during runtime this efficiently(if you ignore the bugs.. :)).
And I had no clue that js-engines has come this far, optimizing code during runtime this efficiently(if you ignore the bugs.. :)).
It's actually really well written
Some of the optimizations compilers/jits will perform would blow your mind.
LICM is a very simple thing to do.
There are more advanced things, like http://www.cs.rice.edu/~keith/512/2011/Lectures/L06CLZ-1up.p...
etc
That's just on the code movement side too.
Most good static compilers can symbolically evaluate even linear systems of equations/recurrences/etc that are solvable, and transform loops that calculate them into constants, etc.
Some of the optimizations compilers/jits will perform would blow your mind.
LICM is a very simple thing to do.
There are more advanced things, like http://www.cs.rice.edu/~keith/512/2011/Lectures/L06CLZ-1up.p...
etc
That's just on the code movement side too.
Most good static compilers can symbolically evaluate even linear systems of equations/recurrences/etc that are solvable, and transform loops that calculate them into constants, etc.
> LICM is a very simple thing to do.
Yeah, it sometimes surprises me how much value the most simple optimizations have and how much they break people's attempts to measure performance.
There is another side to this medal which is best expressed in a quote I picked up from a relatively old paper:
"A survey of the literature on optimization uncovers many techniques that are hard to understand, harder to implement, more general than necessary and of marginal value. Our approach was to do the most profitable machine independent optimizations in the most profitable way" (A Portable Optimizing Compiler for Modula-2 by Michael L. Powel)
In some sense the more microbenchmarks an optimization breaks the bigger its impact on real world code is :)
Yeah, it sometimes surprises me how much value the most simple optimizations have and how much they break people's attempts to measure performance.
There is another side to this medal which is best expressed in a quote I picked up from a relatively old paper:
"A survey of the literature on optimization uncovers many techniques that are hard to understand, harder to implement, more general than necessary and of marginal value. Our approach was to do the most profitable machine independent optimizations in the most profitable way" (A Portable Optimizing Compiler for Modula-2 by Michael L. Powel)
In some sense the more microbenchmarks an optimization breaks the bigger its impact on real world code is :)
While that paper makes a generally true statement, trying to do the most profitable optimizations in the most profitable way is what dead-ended gcc for many many years :)
At some point, you have to step back from what you are doing and actually design and engineer optimizations coherently, not just "do this little thing and that little thing"
At some point, you have to step back from what you are doing and actually design and engineer optimizations coherently, not just "do this little thing and that little thing"
LICM is indeed very simple to do, I wrote it for a JIT emulator I was working on a while ago. It turns out that it's not always fast. There's a bunch of weird cases that happen that can destroy your performance.
LICM is highly dependent on the quality of your trace recording or method inlining.
Not disagreeing with what you wrote at all, just wanted to provide an anecdote.
LICM is highly dependent on the quality of your trace recording or method inlining.
Not disagreeing with what you wrote at all, just wanted to provide an anecdote.
Indeed there are some relatively well understood negative consequences to LICM and various other redundancy elimination optimizations: e.g. they increase life-time of values which can have negative impact on register allocation. Another thing is getting LICM right in the presence of conditional control flow within the loop is a non-trivial exercise, in dynamically typed languages this becomes a problem because JIT in general wants to hoist type guards aggressively but it has to account for conditional control flow to avoid weird/unnecessary deopts
Yes, LCM is pretty much a disaster in practice, and people have been trying to fix it ever since.
The set of things you are worrying about it actually precisely the set of things CLZ solves :)
The set of things you are worrying about it actually precisely the set of things CLZ solves :)
> The set of things you are worrying about it actually precisely the set of things CLZ solves :)
I think the set of things I worry about is slightly different, because they operate in a static environment and I operate in a dynamic one. For example, they don't seem to be worried about the transformation from:
I think the set of things I worry about is slightly different, because they operate in a static environment and I operate in a dynamic one. For example, they don't seem to be worried about the transformation from:
loop {
if (pred) {
// o and o.f are loop invariants
Guard(o is X);
v = Load(o.f);
Use(v)
}
}
Guard(o is X);
v = Load(o.f);
loop {
if (pred) {
Use(v)
}
}
This was a great talk and funny one at that : ) Great perspective on what to really be concerned with. There were other great talks at #GOTOChgo but this one is one of the best ones : )
You should never "optimize" JavaScript code, like objects vs arrays or methods vs functions. But instead try to optimize your algorithms, like leaving out square root when comparing witch line is longer.
I did optimize JavaScript in the past, but it always worked different in different JS implementations and could change in different releases of the same JS engine. So I gave up on it, and it actually makes me more productive not having to think about with method is faster.
I did optimize JavaScript in the past, but it always worked different in different JS implementations and could change in different releases of the same JS engine. So I gave up on it, and it actually makes me more productive not having to think about with method is faster.
That is simply not true. To write fast JS you need to understand the basics of the engine, like what a hidden class is, what exactly garbage collector does, how to keep your methods monomorphic or how types work in arrays. I was stunned when recruiting people to see how many just didn't understand why the prototype chains they build are actually creating a separated method instance per each object they allocate. You can see that easily when serious people suggest that an object constructor should return another object with methods on it. Sure, it's valid JS, it has it's advantages and compiler MAY be able to optimize it (last time I checked it didn't), but there will be a limit in what it can do and performance will be impacted.
You should perhaps not optimize trivial things like methods vs function, because there is no good underlying reason for them to be much different. You should know when to optimize and what. I love the longer version of Knuth's quote - "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.". The overall architecture, the way you write and organize your code, in my opinion, and in case of JavaScript is likely to be in the 3%.
But I also agree not caring too much about it improves productiveness. Most applications really don't need to worry about performance in such depth. Even when they do, it often boils down to keeping one or two core components optimized.
The only worry I always have about it, is that when you notice your code style and architecture doesn't play well with JS engines, it's often too late to change it. A friend called me a day before a prod release asking what to do about his program taking a huge chunk of memory for what the profiler described as a "compiled code". It was a large corporate application, with dynamic code loading. We digged into the framework internals just to notice there was no built-in way of deregistering a class once it's created. Since it didn't play nicely with the JS engine, there were many more hidden classes than business models in the system, and most calls were polymorphic. All this compiled code had to be kept somewhere. At this point there was no way he would replace the framework and even quickly fixing it was a challenge.
You should perhaps not optimize trivial things like methods vs function, because there is no good underlying reason for them to be much different. You should know when to optimize and what. I love the longer version of Knuth's quote - "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.". The overall architecture, the way you write and organize your code, in my opinion, and in case of JavaScript is likely to be in the 3%.
But I also agree not caring too much about it improves productiveness. Most applications really don't need to worry about performance in such depth. Even when they do, it often boils down to keeping one or two core components optimized.
The only worry I always have about it, is that when you notice your code style and architecture doesn't play well with JS engines, it's often too late to change it. A friend called me a day before a prod release asking what to do about his program taking a huge chunk of memory for what the profiler described as a "compiled code". It was a large corporate application, with dynamic code loading. We digged into the framework internals just to notice there was no built-in way of deregistering a class once it's created. Since it didn't play nicely with the JS engine, there were many more hidden classes than business models in the system, and most calls were polymorphic. All this compiled code had to be kept somewhere. At this point there was no way he would replace the framework and even quickly fixing it was a challenge.
> To write fast JS you need to understand the basics of the engine
The problem is that there are many different engines out there.
Unless of course you are talking about server-side code. But for the server-side, there are more efficient solutions than just javascript.
The problem is that there are many different engines out there.
Unless of course you are talking about server-side code. But for the server-side, there are more efficient solutions than just javascript.
Sure they are, but they employ mostly the same techniques. The implementation may differ, but the ideas remain the same, at least to the extent where well organized code optimized for one engine is unlikely to be slow in another.
[deleted]
You should optimize your images. These are 6000x4500px 2.4MB PNG images:
http://mrale.ph/talks/goto2015/images/intro-slide.png
http://mrale.ph/talks/goto2015/images/last-slide.png
http://mrale.ph/talks/goto2015/images/intro-slide.png
http://mrale.ph/talks/goto2015/images/last-slide.png
Oops, sorry for that. Forgot to shrink them before publishing. Will fix as soon as I get to a place with a stable internet connection - traveling right now.
upd. done, scaled down to 1024x768
upd. done, scaled down to 1024x768
[deleted]
His blog http://mrale.ph/ is full of hidden goodies like Object.create vs function constructor [1] and hidden classes vs jsperf [2]. Lots of good stuff on his SO too.
It'd be great if he (hi if you're reading this btw!) could write a bit about TF and the changes being made to v8 internally from an overview point of view.
[1] http://mrale.ph/blog/2014/07/30/constructor-vs-objectcreate.... [2] http://mrale.ph/blog/2013/08/14/hidden-classes-vs-jsperf.htm...