Java vs. Go performance(benchmarksgame.alioth.debian.org)
benchmarksgame.alioth.debian.org
Java vs. Go performance
http://benchmarksgame.alioth.debian.org/u64q/compare.php?lang=java&lang2=go
68 comments
As someone who has a fairly strong distaste for Java: It's not unfair at all. If I want to make the most of my CPU, and Java runs faster than Go, then that's the way it is.
There may be historical reasons for this. There may be low hanging fruit for Go. But in the end, all that matters to the consumer of my project is "code go fast?".
There may be historical reasons for this. There may be low hanging fruit for Go. But in the end, all that matters to the consumer of my project is "code go fast?".
My understanding is that Go has a very simple optimizer. They were waiting for all the C code in the compiler to be translated to Go, which I believe was done in the last release.
not an expert, but i think you are talking about Go's garbage collection, which i don't think is an optimizer.
Some improvement to the garbage collector was done in the last release, and involved moving much of the runtime to Go. Making the compiler into Go, however, is actually only completed in the 1.5 (development) branch, to be released in... August, I think. That'll open up the work on the optimizer (and everything else in the compiler) to be done in Go.
"Fair" to the Go designers? No. Go may indeed improve as time goes by.
"Fair" to companies considering Go right now? Absolutely. If you're deploying a production system, you go to war with the army you have.
"Fair" to companies considering Go right now? Absolutely. If you're deploying a production system, you go to war with the army you have.
To clarify, I was making a point about the implication inherent in a "This vs. That" comparison... IMHO it is a total distraction, especially with contrived benchmarks, no matter how good they are.
A few years back I remember Brian Goetz (I think) wrote in an article that object creation in Sun's JVM was down to like 10 machine instructions (I assume on x86). How much effort will it take to get that down to 9, 8, 7?
On the other hand, choosing a language like Go that (I assume) still has a good deal of room from optimization could over time make applications run faster/better/cheaper without changing a single line of code.
As for the "you go to war with the army you have" idea... that's fine, but by far the biggest performance problems in nearly every application I've seen are the result of programming errors/laziness/deadlines, and there is nothing that any platform can do to optimize away those problems
A few years back I remember Brian Goetz (I think) wrote in an article that object creation in Sun's JVM was down to like 10 machine instructions (I assume on x86). How much effort will it take to get that down to 9, 8, 7?
On the other hand, choosing a language like Go that (I assume) still has a good deal of room from optimization could over time make applications run faster/better/cheaper without changing a single line of code.
As for the "you go to war with the army you have" idea... that's fine, but by far the biggest performance problems in nearly every application I've seen are the result of programming errors/laziness/deadlines, and there is nothing that any platform can do to optimize away those problems
I don't think it is that unfair.
From my perspective the JVM hasn't seen major performance changes since the early days e.g. Java 4. All of the major improvements have come in the library space e.g. LMAX Disruptor, OpenHFT, Javolution.
From my perspective the JVM hasn't seen major performance changes since the early days e.g. Java 4. All of the major improvements have come in the library space e.g. LMAX Disruptor, OpenHFT, Javolution.
That is utter and complete rubbish. Every major Java release has come with significant performance improvements.
What perspective is that?
They ship new optimisations in more or less every release. I'm not sure what workload you do that you've never seen performance improvements.
For example in Java 9 they're going to ship auto-vectorisation for certain kinds of functional / parallel stream construct.
They ship new optimisations in more or less every release. I'm not sure what workload you do that you've never seen performance improvements.
For example in Java 9 they're going to ship auto-vectorisation for certain kinds of functional / parallel stream construct.
Almost exclusively J2EE apps.
I am not saying there weren't any benefits from the JVM over the years only that the biggest gains I've experienced have come from new types of libraries.
Fully agree with Java 9 though. It is the first release that I've been really excited about. Project Jigsaw has the potential to fundamentally reimagine the entire platform.
I am not saying there weren't any benefits from the JVM over the years only that the biggest gains I've experienced have come from new types of libraries.
Fully agree with Java 9 though. It is the first release that I've been really excited about. Project Jigsaw has the potential to fundamentally reimagine the entire platform.
> I'm sure that there is a lot of low hanging fruit for Go to grab at to improve performance, and most likely will.
If potential improvement becomes actual improvement, you'll most likely see that result when the measurements shown on this website are updated for that version of Go.
To-date the measurements for Go have been updated 9 times since Go 1.0 (3 years ago) and another 50 times in the 2 years before Go 1.0.
If potential improvement becomes actual improvement, you'll most likely see that result when the measurements shown on this website are updated for that version of Go.
To-date the measurements for Go have been updated 9 times since Go 1.0 (3 years ago) and another 50 times in the 2 years before Go 1.0.
[deleted]
I find these benchmarks slightly dubious because everytime I look into the actual code I find that is written some obscure manner.
For example - here is the code of the Java "knucleotide" benchmark:
http://benchmarksgame.alioth.debian.org/u64q/program.php?tes...
(If I understand the website correctly.)
Look at the way data is read (scanPastHeader, readRestIntoByteArray, fullSequence).
Or how the result of calculateHash is put into a HashMap where it immediately would be hashed again.
I think this is neither the most natural or performant way of solving the problem in Java.
For example - here is the code of the Java "knucleotide" benchmark:
http://benchmarksgame.alioth.debian.org/u64q/program.php?tes...
(If I understand the website correctly.)
Look at the way data is read (scanPastHeader, readRestIntoByteArray, fullSequence).
Or how the result of calculateHash is put into a HashMap where it immediately would be hashed again.
I think this is neither the most natural or performant way of solving the problem in Java.
Currently 7 Java k-nucleotide programs are shown.
> the most natural or performant way of solving the problem in Java
"How to contribute programs"
http://benchmarksgame.alioth.debian.org/play.html#contribute
> the most natural or performant way of solving the problem in Java
"How to contribute programs"
http://benchmarksgame.alioth.debian.org/play.html#contribute
So submit an improved version?
Interesting. All this time I have a perception that Java app is slow as hell. What a terribly wrong assumption. Color me impressed.
Well, a lot of Java web apps ARE slow as hell, but one can't really blame Java-the-language (or the JVM) for it.
Two examples from my experience:
JSF is doing lots of "magic" trying to keep state. It isn't exactly quick to render a simple hello world page, but if you mess up the state on a somewhat complex page, JSF spends ages in it's six (yes, six!) lifecycle stages [0].
Hibernate is probably the worst offender. Well, the interface is somewhat nice: you don't have to care about loading objects from the database, Hibernate will load them for you. One object, one request at a time. Looking at the hundreds of generated SQL requests it's quite easy to figure out how you would write better SQL... but you don't write the SQL, Hibernate does :)
The lesson IMO is: If you're writing everything from scratch, language performance is important. If you're using frameworks, the performance of the whole stack is way more important and one should probably compare benchmarks - or even real-world applications that are similar to ones use case - of the whole stack.
[0] http://docs.oracle.com/javaee/5/tutorial/doc/bnaqq.html
Two examples from my experience:
JSF is doing lots of "magic" trying to keep state. It isn't exactly quick to render a simple hello world page, but if you mess up the state on a somewhat complex page, JSF spends ages in it's six (yes, six!) lifecycle stages [0].
Hibernate is probably the worst offender. Well, the interface is somewhat nice: you don't have to care about loading objects from the database, Hibernate will load them for you. One object, one request at a time. Looking at the hundreds of generated SQL requests it's quite easy to figure out how you would write better SQL... but you don't write the SQL, Hibernate does :)
The lesson IMO is: If you're writing everything from scratch, language performance is important. If you're using frameworks, the performance of the whole stack is way more important and one should probably compare benchmarks - or even real-world applications that are similar to ones use case - of the whole stack.
[0] http://docs.oracle.com/javaee/5/tutorial/doc/bnaqq.html
Well, to be fair, Hibernate is just a tool - and as many tools, it can be used the wrong way and the right way. Most people seem to not spend any time to learn how to use it, and then complain when it doesn't magically read their mind. It's leaky, like all abstractions. You have to understand how it works, and how databases works. There's always the option to write HQL or JPQL as well, or even fall back to SQL if all else fails.
It's not Hibernate's fault that you eager load everything, always load entire entities when you just need a single boolean field, or loop over entity collections to aggregate values - Hibernate is just a tool, and it's the developers who are telling it what to do.
It's not Hibernate's fault that you eager load everything, always load entire entities when you just need a single boolean field, or loop over entity collections to aggregate values - Hibernate is just a tool, and it's the developers who are telling it what to do.
true, but... if so many developers end up having issues on not-trivial-hello-world-employee-customer example, then maybe, but just maybe it's not the best tool for most. I mean, it's around for ages, the principle didn't change a bit, and same issues again and again.
One thing that I don't like about Hibernate - most projects evolve, for a very long time even after delivery. Messing with OR mappings (which most changes do) can include a lot of regression on performance side, much more than SQL (unless you do something horrible on DB level). Also, the more complex data model is (and often you don't choose how it's done), the less benefits you get from solutions like these.
One thing that I don't like about Hibernate - most projects evolve, for a very long time even after delivery. Messing with OR mappings (which most changes do) can include a lot of regression on performance side, much more than SQL (unless you do something horrible on DB level). Also, the more complex data model is (and often you don't choose how it's done), the less benefits you get from solutions like these.
What's wrong with having six lifecycle stages? I don't use JSF (or Java) but I can understand the usefulness of having several lifecycle methods assuming you can hook into them and add your own custom logic. The more lifecycle stages the more precisely you can choose when during the request processing your custom logic should be invoked.
You're right, there isn't anything wrong with the six stages per se.
The root cause is really that (a) they're forcing a stateful model onto the web and (b) that the way they're keeping/restoring the state is expensive.
The lifecycle stages are more a sympton of that. Most web framework these days go with a stateless approach. One notable exception is the lift framework for scala (although I'm not sure if it's really active anymore). I have only looked very briefly at it, but it seemed to give the programmer more explicit control of the state...
A simplified example to demonstrate the problem: Imagine a B2B app with a medium-sized table, where each row contains multiple objects. A HTTP request is send for performing an action on one object.
In a stateless approach, the programmer decides which objects he needs to serve the request. In JSF, the framework restores the complete state as seen in the previous request, and will also setup a lot of other magic (event handlers, validators, what not) - no matter whether it is actually needed to serve the request or not.
So one ends up with stages in the lifecycle that waste a lot of time, but one really hasn't much control over.
The root cause is really that (a) they're forcing a stateful model onto the web and (b) that the way they're keeping/restoring the state is expensive.
The lifecycle stages are more a sympton of that. Most web framework these days go with a stateless approach. One notable exception is the lift framework for scala (although I'm not sure if it's really active anymore). I have only looked very briefly at it, but it seemed to give the programmer more explicit control of the state...
A simplified example to demonstrate the problem: Imagine a B2B app with a medium-sized table, where each row contains multiple objects. A HTTP request is send for performing an action on one object.
In a stateless approach, the programmer decides which objects he needs to serve the request. In JSF, the framework restores the complete state as seen in the previous request, and will also setup a lot of other magic (event handlers, validators, what not) - no matter whether it is actually needed to serve the request or not.
So one ends up with stages in the lifecycle that waste a lot of time, but one really hasn't much control over.
Oh sure, the stateful model thing you mention definitely sucks which is why I hate using asp.net web forms and prefer asp.net mvc at the office.
Both of them have lifecycle methods, (for app start, request start, request end, etc.) and for the most part I completely ignore them until the day I need to use them and I'm glad they exist because the alternative would have been writing a code in every endpoint of my application.
Both of them have lifecycle methods, (for app start, request start, request end, etc.) and for the most part I completely ignore them until the day I need to use them and I'm glad they exist because the alternative would have been writing a code in every endpoint of my application.
[deleted]
You don't have to use all Hibernate features. You can write your own SQLs too if you want to.
Hibernate can be super fast if you use the correct strategies [1] at your bottlenecks. Any other performance issues are typically ORM related.
[1]: http://docs.jboss.org/hibernate/core/3.3/reference/en/html/p...
[1]: http://docs.jboss.org/hibernate/core/3.3/reference/en/html/p...
Any Java program takes a while to start up, and interactive Java desktop apps tend to be slow because of the UI toolkits they use. But in terms of throughput for a backend service, Java performance is fantastic.
Once you heat up the JVM, Java can run like a bat out of hell. And, if you don't run into GC problems, it's as fast as anything out there.
You can't fine-tune memory usage and can run into cache issues (or worse) if you deal with quite a lot of data, though.
Wait. What ? I have never seen any piece of technology with more tuning options than the JVM. Especially given how dramatically different your apps perform with different combinations. It's akin to old wifes tales about which settings to use when.
http://www.oracle.com/technetwork/articles/java/vmoptions-js...
And don't forget that much of big data is on the JVM e.g. Hadoop, Cassandra, HBase.
http://www.oracle.com/technetwork/articles/java/vmoptions-js...
And don't forget that much of big data is on the JVM e.g. Hadoop, Cassandra, HBase.
> You can't fine-tune memory usage
What do you mean by that? There are some very fine-grained tuning options if you really need them.
What do you mean by that? There are some very fine-grained tuning options if you really need them.
Maybe I'm wrong, but I don't know how you can say you want, say, a list of objects of class Foo be in contiguous slots of memory, rather than having a list of references to objects. Because the latter is very cache-unfriendly. In go, you can say whether you want a []Foo or a []*Foo
You can do that "by hand" with buffers (and the HFT guys do) but yeah it's not easy or fun. C#-style struct types are supposedly coming in the next version.
That comes from the fact that Java takes quite a while to startup. It is going to be hopefully fixed in Java 9 where the platform becomes more modular. But that issue is what caused Java to have such a terrible reputation when apps were run on the desktop.
Probably part of it, but for me, the somewhat uncanny valley-ish non-standard look and behaviour of Swing UIs (and AWT before that) is the biggest turn off. Using java apps on the desktop feels a little like using a keyboard with mittens.
This is an issue of unskilled devs using ancient UI design approach, usually not even bothering with skinning it.
Things like bitTorrent were done in Java (correct me if I'm wrong), and you have no clue which language they are done in when looking at UI. But this a bit of extra effort, and for intranet apps, nobody usually bothers.
Another reason for oh-not-so-windows-look of UI is something microsoft finally begun discovering - that there are other platforms than Windows. Java UI works anywhere where you have working JRE, and if you've ever seen Linux's KDE/GNOME/XFCE etc. then you might realize it ain't easy to come up with a cross-section between them.
You need to hack things with Wine or similar to have it +-++ work the other way around (ie MFC app -> anywhere but MS world, and even there it might be tricky with all dll hell).
Things like bitTorrent were done in Java (correct me if I'm wrong), and you have no clue which language they are done in when looking at UI. But this a bit of extra effort, and for intranet apps, nobody usually bothers.
Another reason for oh-not-so-windows-look of UI is something microsoft finally begun discovering - that there are other platforms than Windows. Java UI works anywhere where you have working JRE, and if you've ever seen Linux's KDE/GNOME/XFCE etc. then you might realize it ain't easy to come up with a cross-section between them.
You need to hack things with Wine or similar to have it +-++ work the other way around (ie MFC app -> anywhere but MS world, and even there it might be tricky with all dll hell).
Standard bittorrent was in Python. Azureus was in Java and it felt significantly slower (than Python!) - it took so long to start it needed a splash screen, and you'd click a menu and wait for it to appear.
Kind of an unfair comparison - the Python one just calls native Qt or Tk whereas the Java UI is actually written in Java - but it does colour the perception.
Kind of an unfair comparison - the Python one just calls native Qt or Tk whereas the Java UI is actually written in Java - but it does colour the perception.
Startup time isn't that bad. It takes about 500 msec to get a window on the screen with the latest versions of Java and the JavaFX toolkit. Most of that is UI init. If you're doing command line apps it's much faster.
There's a series of articles about how Java don't have to be slow XML-laden crap: http://blog.paralleluniverse.co/2014/05/01/modern-java/
While I don't necessarily agree with everything in it (like Gradle builds), it's interesting to look at.
While I don't necessarily agree with everything in it (like Gradle builds), it's interesting to look at.
If you don't like XML but don't want to switch to Gradle, you could use another language atop Maven: https://github.com/takari/polyglot-maven
Go has a significantly smaller memory usage in all of the benchmarks, often by a factor of 10x. For most workloads, this can have a huge impact on when you need to spend more money to support your business.
"Huge differences in default memory allocation don't necessarily mean there'll be huge differences in Memory-used for tasks that require memory to be allocated."
http://benchmarksgame.alioth.debian.org/play.html#whymemory
http://benchmarksgame.alioth.debian.org/play.html#whymemory
I was thinking the same thing. It depends of your needs, if you need raw performance and you have at your disposition a lot of RAM on a powerful server , Java might be a good bet. If you are building things for a normal computer, a Raspberry PI or a background service, the memory usage of Java is going to hit you badly.
well, you don't need to run a virtual machine, so...
Since we're posting benchmarks, here's Java vs Haskell: http://benchmarksgame.alioth.debian.org/u64q/compare.php?lan.... The Haskell implementations generally use much less memory than the Java ones and perform competitively. In spite of criticism that some of the Haskell implementations are not idiomatic, they're no more verbose than the Java ones in terms of LOC.
This benchmark is out of date almost immediately because of the patch level of the Java runtime and the Golang compilers change frequently.
I was half thinking of creating a VB6 module for LLVM just as a tongue in cheek joke of taking a language with a slow runtime and effectively strapping a V8 engine to it, but I saw someone already did most of the work: https://github.com/microcai/llvm-qbasic
I was half thinking of creating a VB6 module for LLVM just as a tongue in cheek joke of taking a language with a slow runtime and effectively strapping a V8 engine to it, but I saw someone already did most of the work: https://github.com/microcai/llvm-qbasic
For that to be an argument two things would be required:
1) Changing patchlevels causing huge performance differences. 2) Those "patchlevels" being immediately deployable to production in the real world.
Neither is true, so you don't have an argument.
1) Changing patchlevels causing huge performance differences. 2) Those "patchlevels" being immediately deployable to production in the real world.
Neither is true, so you don't have an argument.
> This benchmark is out of date almost immediately
Why do you think that matters?
Why do you think that matters?
This is a pretty eye opening chart. I know the Go guys have a philosophical point about wanting to be able to reason the whole source code as the justification for the Plan 9 and later GCC only stance, but I do have to think that these numbers are poor enough that a LLVM port should be investigated.
I also think that it's a good think for Go that Apple hasn't open source'd swift. I suspect it would thrash Go here.
I also think that it's a good think for Go that Apple hasn't open source'd swift. I suspect it would thrash Go here.
Maybe I come from a different world, but these numbers seem pretty good. If you compare C++ and Go, C++ is 3x-1x faster than Go using more code.
For most applications that you'd expect to use Go for, that's decent efficiency at a smaller memory footprint than Java. Enough, certainly, to make a high-performance server that uses less RAM than Java and less code than C++.
For most applications that you'd expect to use Go for, that's decent efficiency at a smaller memory footprint than Java. Enough, certainly, to make a high-performance server that uses less RAM than Java and less code than C++.
Go seems to be heavily optimized for write-time efficiency instead of runtime efficiency.
There is only one where Go is 3 times slower, regex-dna. That would either be because of the speed of the regex implementation or the GC for the allocations. Apart from that Go is similar or twice as fast for reverse-component. Note that Go uses less memory for tasks that require memory to be allocated too.
Go is doing better at the benchmarks game than it has in the past.
Go is doing better at the benchmarks game than it has in the past.
I don't get it - is this because you think Java has poor performance?
I for one suspect that Swift does not exceed Java in throughput either and in fact I'd bet that the performance of Swift is worse.
I for one suspect that Swift does not exceed Java in throughput either and in fact I'd bet that the performance of Swift is worse.
As I understand it, Go also give some guarantees about the maximum amount of time the garbage collection will stall the program. An issue where ever you are trying to achieve consistent performance. As I understand it most of the java benchmarks postpone clearing up memory until it can be done trivially at exit, something you just can't do in a whole class of programs.
Not yet, but that is planned for the next release.
The way I read the results Java is basically faster only on the regex benchmark and it is known fact that today go has a slow regex library. Everything else the results show Java to be slower with larger code and larger memory footprint.
Why are all the Java fans psyched by these results?
Why are all the Java fans psyched by these results?
> faster only on the regex benchmark
fwiw Also binary-trees, pidigits, n-body.
http://benchmarksgame.alioth.debian.org/u64q/compare.php?lan...
fwiw Also binary-trees, pidigits, n-body.
http://benchmarksgame.alioth.debian.org/u64q/compare.php?lan...
Now if only the benchmarks game would actually support, say, PyPy. Or LuaJIT.
But nope, one implementation per language, and "language" defined rather arbitrarily at that.
But nope, one implementation per language, and "language" defined rather arbitrarily at that.
Please take the program source code and the measurement scripts and publish your own measurements.
afaict we all feel the same way about this, we all feel that we should sit on our hands and wait for someone else to do the chores we don't wish to do.
http://benchmarksgame.alioth.debian.org/play.html#languagex
afaict we all feel the same way about this, we all feel that we should sit on our hands and wait for someone else to do the chores we don't wish to do.
http://benchmarksgame.alioth.debian.org/play.html#languagex
I would, if I had access to a machine that was somewhat stable w.r.t. performance.
Alas, my only machine currently is my laptop, and it overheats often enough on stress-tests that it's... less than accurate, shall I say?
Alas, my only machine currently is my laptop, and it overheats often enough on stress-tests that it's... less than accurate, shall I say?
There are also many options available to compare different languages. Very cool tool.
Also you can dive-into the source code and see the different ways people have written programs in the same language --
http://benchmarksgame.alioth.debian.org/u64q/performance.php...
Also for an overview --
http://benchmarksgame.alioth.debian.org/u64/which-programs-a...
Also for a different overview --
http://benchmarksgame.alioth.debian.org/u64/code-used-time-u...
http://benchmarksgame.alioth.debian.org/u64q/performance.php...
Also for an overview --
http://benchmarksgame.alioth.debian.org/u64/which-programs-a...
Also for a different overview --
http://benchmarksgame.alioth.debian.org/u64/code-used-time-u...
That last link...very very cool. Much more interesting than the Java vs go link I posted. You should submit it and spark a discussion :)
By my guest.
It's probably been submitted several times over the years, but there are always people who haven't seen the benchmarks game at all, or have only seen the direct comparisons.
The interesting unique detail is "Shortest C++".
It's probably been submitted several times over the years, but there are always people who haven't seen the benchmarks game at all, or have only seen the direct comparisons.
The interesting unique detail is "Shortest C++".
I wonder if there will be any improvement if the Go code is compiled with GCC.
But this is kind of unfair I think... the OpenJDK JVM has 20 years of development behind it, and has probably had mountains of money sunk into tuning f* out of it by the biggest most enterprisy players in the game.
I'm sure that there is a lot of low hanging fruit for Go to grab at to improve performance, and most likely will. I can't stand benchmarks like this.
Oh yeah, and Java FTW ;)