Web Latency Benchmark: A new kind of browser benchmark(google.github.io)
google.github.io
Web Latency Benchmark: A new kind of browser benchmark
http://google.github.io/latency-benchmark/
17 comments
Awesome to see a non-Virtual Reality use of our Latency Tester.
It's an awesome piece of hardware, thank you for making it! I'm currently brainstorming ways to use it more extensively in the tests.
This benchmark does have a connection to virtual reality, as part of my inspiration to make it came from my experience using vr.js[1] and WebGL for virtual reality in browsers[2]. The Web isn't yet ready for VR due to latency issues, but I hope to change that!
On that note, are you guys interested in browser support for the Rift? I believe the latency issues are surmountable with effort and some cooperation from browser vendors, and I'd love to see something official in that direction (in fact, I'd love to work on something official).
[1] https://developer.oculusvr.com/forums/viewtopic.php?f=20&t=7...
[2] This is what I made: http://www.youtube.com/watch?v=pWYlTIt4j0Y&hd=1&t=1m11s
This benchmark does have a connection to virtual reality, as part of my inspiration to make it came from my experience using vr.js[1] and WebGL for virtual reality in browsers[2]. The Web isn't yet ready for VR due to latency issues, but I hope to change that!
On that note, are you guys interested in browser support for the Rift? I believe the latency issues are surmountable with effort and some cooperation from browser vendors, and I'd love to see something official in that direction (in fact, I'd love to work on something official).
[1] https://developer.oculusvr.com/forums/viewtopic.php?f=20&t=7...
[2] This is what I made: http://www.youtube.com/watch?v=pWYlTIt4j0Y&hd=1&t=1m11s
1.1 out of 10 running the latest stable version of Chrome on a 2010 13" MacBook Pro with a 2.40GHz Core 2 Duo, 8GB of RAM, and a Samsung 840 SSD. Geez... Is there anything I can do to improve the score?
odd, i have much the same mac, same cpu/ram with a regular 7200rpm disk, same chrome and i get 2.5
Has anyone assembled benchmarks across the different browsers yet? I like the aims of this benchmark a lot -- I've recently switched away from Firefox to Chrome simply because it feels less janky.
(benchmark author here) I don't have comprehensive numbers compiled yet, but I can give you a general ranking of browsers. From best score to worst, the current ranking is roughly:
Safari
Firefox (except Mac)
IE 11
Chrome and Mac FirefoxI wonder if I'm using my browser differently than others or if there is something wrong with my machine. Whenever I try to switch from Chrome to FF it seems way behind Chrome in terms of speed and snappiness.
edit:
Windows 8.1, SSD, i3, 16GB RAM (63% used)
Chrome 30.0.1599.101 m: 3.4/10
Chrome Incognito: 4/10
Firefox 25: 5.4/10
FF Private Browsing 3/10 O.o
All tests done at least twice with similar numbers.
But Chrome has been running for a few days while FF was just started. Guess I'll leave FF running a bit and see if it changes anything.
Also interesting, FF takes around 2-3 times as long to run the tests than Chrome.
The reason for the low Chrome score is this:
JavaScript jank Tests responsiveness during JavaScript execution. CSS: 1.0 frames jank, Scrolling: 170.1 frames jank
edit:
Windows 8.1, SSD, i3, 16GB RAM (63% used)
Chrome 30.0.1599.101 m: 3.4/10
Chrome Incognito: 4/10
Firefox 25: 5.4/10
FF Private Browsing 3/10 O.o
All tests done at least twice with similar numbers.
But Chrome has been running for a few days while FF was just started. Guess I'll leave FF running a bit and see if it changes anything.
Also interesting, FF takes around 2-3 times as long to run the tests than Chrome.
The reason for the low Chrome score is this:
JavaScript jank Tests responsiveness during JavaScript execution. CSS: 1.0 frames jank, Scrolling: 170.1 frames jank
There are many different ways to measure speed. For example this benchmark says nothing about page load times or JavaScript speed; Chrome might (hypothetically) be way faster than Firefox there and still score worse in this test.
> FF takes around 2-3 times as long to run the tests than Chrome.
There are a couple of reasons for this. Firstly, lower latency is more difficult to measure, so it takes more trials to get the same number of good measurements. Secondly, Firefox has longer scroll momentum than Chrome, and the benchmark waits for the scroll momentum to die down between each scroll latency measurement.
> FF takes around 2-3 times as long to run the tests than Chrome.
There are a couple of reasons for this. Firstly, lower latency is more difficult to measure, so it takes more trials to get the same number of good measurements. Secondly, Firefox has longer scroll momentum than Chrome, and the benchmark waits for the scroll momentum to die down between each scroll latency measurement.
The main problem I have with FF is it's scrolling performance ;)
Thanks for the explanation, makes a lot of sense.
Can you explain the extreme FF private browsing results?
Thanks for the explanation, makes a lot of sense.
Can you explain the extreme FF private browsing results?
I have no idea why a private browsing window would perform differently, but you're right, I get the same results myself. That's weird! It's exactly the sort of issue I hope to expose with this test. Latency issues like that are often ignored because they're not visible, and I want to make them visible.
Thanks very much for making this test. I use just about every browser out there, and love Chrome on my new Nexus 5, but I've felt that with iOS 7 and 10.9, Safari made huge improvements in responsiveness and it was hard to pin that down until now. I look forward to future optimizations. :)
Is that with a fresh Firefox profile? You could try resetting it to see if that helps: https://support.mozilla.org/en-US/kb/reset-preferences-fix-p... . For the sake of fairness you could do the same to Chrome: https://support.google.com/chrome/answer/3296214?hl=en
No, that was a "real-world" test. It's the FF I had set up after the release of 25 to make yet another try at switching from Chrome.
I can't help but think this should be a test for websites, not computers. It's individual sites full of 3rd party shit that are slow and you can see the performance penalty by installing Ghostery.
It's more of a test for browsers than computers. You can compare Chrome vs. Firefox vs. IE vs. Safari.
Measuring the input latency of individual websites would be interesting too. I'd love to incorporate that into the benchmark if I could figure out a way to do it.
Measuring the input latency of individual websites would be interesting too. I'd love to incorporate that into the benchmark if I could figure out a way to do it.
There are already good tools for measuring network delays. Firefox's web inspector does a beautiful job of graphing how the various requests contribute to overall page load time. This tool is about measuring how responsive the UI is, even when you're still fetching more data.
1.8/10 on a Dell Latitude laptop.
Did anyone else have difficulty getting it to work on a multi-monitor setup? It gave me an error about trying to find the reference image and suggested I switch monitors, but that didn't work. I finally had to disconnect my secondary monitor altogether before it would work.
Did anyone else have difficulty getting it to work on a multi-monitor setup? It gave me an error about trying to find the reference image and suggested I switch monitors, but that didn't work. I finally had to disconnect my secondary monitor altogether before it would work.
(benchmark author here) Interesting. I will have to do more testing on Windows multi-monitor setups. I would not be surprised to learn that it's broken.
If it helps at all, I'm on Windows 7 with my laptop screen as my primary monitor and an LCD screen to the left as my secondary.
I was initially in Win7's "enhanced graphics" mode (can't remember actual feature name) but when I tried to run the benchmark I was thrown into Basic mode. I had a fair amount of other applications running, though, so I may have just run out of memory.
I was initially in Win7's "enhanced graphics" mode (can't remember actual feature name) but when I tried to run the benchmark I was thrown into Basic mode. I had a fair amount of other applications running, though, so I may have just run out of memory.
Being thrown into Basic mode is unfortunately expected. I'll bet the reason it didn't work is having the secondary monitor on the left. I think I only tested it with the secondary monitor on the right.
Multiple monitors. Windows 7. Just got a black screen in both IE9 and Chrome Canary.
Low latency is more important than overall throughput when it comes to UX, so that is a measurement that makes a lot of sense.
Has anyone experience with it?
I learned a new definition for jank today!
3.0/10 on MPBR early 2013 w 16GB and SSD.
Authors should consider a way to collect and present the data.
3.0/10 on MPBR early 2013 w 16GB and SSD.
Authors should consider a way to collect and present the data.
The windows version disables the "Aero Glass" transparency effects on my Win7. A first glance looks like it might not be intentional, but it triggers some kind of compatibility mode in the OS. Not sure why a headless daemon would mess with transparency effects.
Benchmark author here. This is unfortunately intentional due to the way the benchmark works. Measuring latency requires taking screenshots really fast, and on Windows 7 it is very difficult to impossible to take fast screenshots with Aero enabled. In Windows 8 they added a new DirectX screenshot API[1] that makes this unnecessary (which is good because it is impossible to disable Aero on Windows 8).
[1] http://msdn.microsoft.com/en-us/library/windows/desktop/hh40...
[1] http://msdn.microsoft.com/en-us/library/windows/desktop/hh40...
Getting different scores when the test is ran at different browser sizes.
3.4 when the window is sized to it's smallest size.
2.5 when it's at maximum resolution (1600x1200).
Running on OSX 10.9.
3.4 when the window is sized to it's smallest size.
2.5 when it's at maximum resolution (1600x1200).
Running on OSX 10.9.
Yes, window size can definitely affect latency and jank. The benchmark page has a hidden full-window-sized layer that the browser must paint every frame, and a large browser window requires more CPU and GPU work to paint.
Keeps failing for my Win7x64/30.0.1599.101 m.
> Failed to find test pattern on screen. Ensure that your browser's zoom level is set to "100%", and the top-left corner of the window is visible. If you have multiple displays, try moving the browser window to the main display.
* Zoom 100%: check.
* Main monitor (of 2): check.
* Top-left visible: check.
I give up.
> Failed to find test pattern on screen. Ensure that your browser's zoom level is set to "100%", and the top-left corner of the window is visible. If you have multiple displays, try moving the browser window to the main display.
* Zoom 100%: check.
* Main monitor (of 2): check.
* Top-left visible: check.
I give up.
Sorry to hear that. Due you have the Chrome flag for "Experimental Web Platform features" enabled? I recently learned that there's an issue with the test related to this flag.
Edit: It's more likely that it's just buggy support for multiple monitors. I will look into fixing it; for now the only workaround would be disabling your second monitor.
Edit: It's more likely that it's just buggy support for multiple monitors. I will look into fixing it; for now the only workaround would be disabling your second monitor.
Sorry for the late reply; I did notice a Github issue for it so I commented there. Thanks!
Does not work in Safari on Mavericks using a rMBP @1680x1050 retina resolution. It suggests browser zooming be set to 100% which it is.
Thanks for the report; I see the problem. I'll look into fixing it, but in the meantime you can get it running by enabling WebGL in Safari (under the Develop menu, and if you don't have a Develop menu you can enable it in Preferences->Advanced).
Spoiler: Safari actually gets a very good score (relative to other browsers).
Spoiler: Safari actually gets a very good score (relative to other browsers).
I also get "Failed to find test pattern on screen." in Safari with WebGL enabled on a Mid 2011 Mac Mini running 10.9 with dual monitors. View is set to "Actual Size", and I've tried moving the Safari window between monitors.
Other scores - Chrome 30: 2.4/10. Firefox 24 and 25: 1.2/10.
Other scores - Chrome 30: 2.4/10. Firefox 24 and 25: 1.2/10.
Enabling WebGL works. My three tests scored 8.7, 6.7 and 6.3 respectively. I guess Safari, like me, hates repetitive tasks. :)
:) There is a lot of run-to-run variance. Some of it is real in that browsers do give different results, and some of it is just test noise. I'm still working on improving the robustness. Recently I've been working on hardware latency measurement which gives more reliable numbers.
I do have WebGL enabled, and it can't pick up the test pattern.
keep on getting https://github.com/google/latency-benchmark/blob/master/src/... on the first test... not sure why.
Has anyone else heard the term jank before? It's the first time I've seen it.
There is a website [1] with more informations and talks about "jank".
[1] - http://jankfree.org/
[1] - http://jankfree.org/
I use/have used the term 'janky' before to describe similar kinds of things or things that seem off or strange.
Grew up in the midwest fwiw
Grew up in the midwest fwiw
(I'd guess it's originally related to the concept of "jerk" in physics. http://en.wikipedia.org/wiki/Jerk_%28physics%29 )
It's been used in this context since at least 2010 by the Chrome team: http://en.wiktionary.org/wiki/Talk:janky
It's been used in this context since at least 2010 by the Chrome team: http://en.wiktionary.org/wiki/Talk:janky
The Firefox team definitely used the term with this usage, since at least 2011.
We (at Empirical) use janky to describe animations that aren't butter as this author uses it. Not sure how standard but I suspect many people (native speakers) will understand when in context.
No, usually the term I have heard used for this is jitter.
3.3 on a beefy Windows 8 machine running Chrome 31.0.1650.39 beta-m. Interestingly, running an Incognito tab with no extensions allowed to run in incognito results in a 2.7.
Huh.
Huh.
Lower scores are better since we are measuring latency?
Looking at the source, higher scores are better. There are a bunch of tests, and each one produces a score from 0 (bad) to w (good) for some w. The total score is the total of all these.
Given that all the other numbers on the page are ones for which smaller is better, I think the main score itself should be accompanied by a note saying explicitly that higher scores are better.
Given that all the other numbers on the page are ones for which smaller is better, I think the main score itself should be accompanied by a note saying explicitly that higher scores are better.
Good point. I'll add a note next to the scores.
I managed to get a 6.5 on Safari on Mavericks on a 3.5 year old MBP. All the other browser I've tried were below 2, except for Opera Next (2.2).
2.8 / 10.0 on a 1 week old 15" MBP (high-end) running latest stable Chrome.
Somehow the second test always timeouts in IE 11 on Windows 8.1 for me.