Benchmarking D vs Go vs Erlang vs C for MQTT broker(atilanevesoncode.wordpress.com)
atilanevesoncode.wordpress.com
Benchmarking D vs Go vs Erlang vs C for MQTT broker
https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/
52 comments
Erlang's VM is very sophisticated and mature, it is at least years ahead of Go, imo. For example, compare Erlang's pre-emptive scheduling [1] to "goroutines".
[1] http://jlouisramblings.blogspot.com/2013/01/how-erlang-does-...
[1] http://jlouisramblings.blogspot.com/2013/01/how-erlang-does-...
> it is at least years ahead of Go,
Yes, Erlang has admittedly had decades to improve the scheduler (versus a couple of years for the gc compiler in Go), so this is quite literally true.
However, you should know that goroutines are pre-emptively scheduled now; the "scare quotes in your comment seem to imply that they aren't: http://golang.org/doc/go1.2#preemption
Yes, Erlang has admittedly had decades to improve the scheduler (versus a couple of years for the gc compiler in Go), so this is quite literally true.
However, you should know that goroutines are pre-emptively scheduled now; the "scare quotes in your comment seem to imply that they aren't: http://golang.org/doc/go1.2#preemption
I'm a go fanboi, but to be fair, goroutines are now more preemptive, but not completely.
In practice that means that if you call a function somewhere in your infinite loop, you'll get preempted. It's still possible for a single goroutine to steal an os thread.
Practically speaking though, I feel like nobody should be scared away from go due to it's lack of full pre-emption. Seriously, don't write an infinite loop with no function calls.
In practice that means that if you call a function somewhere in your infinite loop, you'll get preempted. It's still possible for a single goroutine to steal an os thread.
Practically speaking though, I feel like nobody should be scared away from go due to it's lack of full pre-emption. Seriously, don't write an infinite loop with no function calls.
[deleted]
Perhaps even more impressive is that it was written by an Erlang novice. This makes me want to hit the books and start learning Erlang again.
He's an Erlang novice, but the guy is the best programmer I've met. I have no doubts that if he chose to pick up D instead he'd've written code as fast as mine.
i see no mention of "concurrency" in that article. i see that the go programs make heavy use of goroutines so why were they run in "single threaded" mode? just testing their mqttsrv with various levels of GOMAXPROCS rewards with immediate benefits. here are some numbers for pingtest and loadtest in Go for varying levels of GOMAXPROCS for them and the server:
(pingtest/mqtt gomaxprocs) vs pingtest, reporting "elapsed time":
as always with benchmarks, caveat emptor, but in my opinion go does just fine.
(pingtest/mqtt gomaxprocs) vs pingtest, reporting "elapsed time":
1x1 5.8s
2x1 6.0s
4x1 6.0s
8x1 6.1s
16x1 6.1s
1x2 5.5s
2x2 3.5s
4x2 3.4s
8x2 3.5s
16x2 3.5s
1x4 5.4s
2x4 3.6s
4x4 3.3s
8x4 3.4s
16x4 3.7s
1x1 75521
2x1 76523
4x1 77477
8x1 74613
16x1 73678
1x2 71439
2x2 100571
4x2 93426
8x2 108938
16x2 97593
1x4 71390
2x4 87426
4x4 98746
8x4 99320
16x4 119692
1x8 64413
2x8 87445
4x8 102953
8x8 120720
16x8 106842
as always with benchmarks, caveat emptor, but in my opinion go does just fine.
Are your numbers in line with the blog's table?
So you get those instead of these?
Loadtest:
Go: 90.9 +/- 11 100.1 +/- 0.1 99.3 +/- 0.2 98.8 +/- 0.3
If not can you make it relevant to the table so people can compare and contrast?
So you get those instead of these?
Loadtest:
Go: 90.9 +/- 11 100.1 +/- 0.1 99.3 +/- 0.2 98.8 +/- 0.3
If not can you make it relevant to the table so people can compare and contrast?
loadtest for 100 500 750 1000, X*Y below means X = loadtest GOMAXPROCS and Y = mqttsrv GOMAXPROCS:
1x1 80735 67696 67068 66585
2x1 64380 76531 31346 43268
4x1 77613 73794 72759 43263
8x1 77813 64197 32899 46960
1x2 71641 72521 72715 72695
2x2 103711 102944 104210 39619
4x2 73171 23803 90537 46204
8x2 96624 49698 33714 45729
1x4 69418 74088 34364 38935
2x4 88805 102847 113740 108829
4x4 102406 108110 34672 120161
8x4 109022 124726 121642 46765
1x8 62336 68238 71297 68754
2x8 104833 115276 118170 44617
4x8 122292 126075 126101 121306
8x8 108826 125098 124765 47713
Still not following. For Loadtest:
D: -- [121.7 +/- 1.5] -- [166.9 +/- 1.5] -- [171.1 +/- 3.3] -- [167.9 +/- 1.3]
C: -- [106.1 +/- 0.8] -- [122.4 +/- 0.4] -- [95.2 +/- 1.3] -- [74.7 +/- 0.4]
E: -- [104.1 +/- 2.2] -- [124.2 +/- 5.9] -- [117.6 +/- 4.6 -- [117.7 +/- 3.2]
G: ---- [90.9 +/- 11] -- [100.1 +/- 0.1] -- [99.3 +/- 0.2] -- [98.8 +/- 0.3]
G(2): ? ? ? ? ?
G(4): ? ? ? ? ?
G(8): ? ? ? ? ?
D: -- [121.7 +/- 1.5] -- [166.9 +/- 1.5] -- [171.1 +/- 3.3] -- [167.9 +/- 1.3]
C: -- [106.1 +/- 0.8] -- [122.4 +/- 0.4] -- [95.2 +/- 1.3] -- [74.7 +/- 0.4]
E: -- [104.1 +/- 2.2] -- [124.2 +/- 5.9] -- [117.6 +/- 4.6 -- [117.7 +/- 3.2]
G: ---- [90.9 +/- 11] -- [100.1 +/- 0.1] -- [99.3 +/- 0.2] -- [98.8 +/- 0.3]
G(2): ? ? ? ? ?
G(4): ? ? ? ? ?
G(8): ? ? ? ? ?
I mentioned concurrency when I said that using more than one system thread didn't help. I tried varying GOMAXPROCS. It either had no effect or made everything slower. And yes, the defaults for loadtest are useless, they need to run for at least a few seconds.
Your results aren't the same ones I got. Maybe it's different with Go 1.2. I have to go and check.
the server (mqttsrv) needs GOMAXPROCS>1 for any appreciable changes in the results, otherwise we're just measuring contention in the client.
i ran on a 2-core intel desktop running linux.
i ran on a 2-core intel desktop running linux.
"the Erlang version I’m not sure"
At least try to find out if it's HiPE (native compilation) or not?
At least try to find out if it's HiPE (native compilation) or not?
"an excuse to learn Erlang, and wrote his own implementation"
Why even included Erlang if you're not going to share the source...
Why even included Erlang if you're not going to share the source...
He wasn't in the office. I'm going to ask him about it today. Like it says at the end of the blog post, if he puts it up on bitbucket I'll include the link.
There's a link to it now.
"the Go implementation was executed with go run ". I think it would be much more appropriate to build the GO script with "go build" and use the resultant binary.
Because the D program was compiled.
http://golang.org/src/cmd/go/run.go?ModPagespeed=noscript
"Run compiles and runs the main package comprising the named Go source files."
"Run compiles and runs the main package comprising the named Go source files."
[deleted]
I found this article a little hard to read;
- #555 on #fff is distinctly too light for a web page's text color (I generally recommend around #202020);
- the font-size of 13px is too small—remove it and let the default of 16px show.
- #555 on #fff is distinctly too light for a web page's text color (I generally recommend around #202020);
- the font-size of 13px is too small—remove it and let the default of 16px show.
"Go is an opinionated language, which would be fine if I agreed with its creators’ opinions." Spot on.
Would have been cool to see how many lines of code each implementation took including the erlang one.
It's mentioned. It's not easy to measure but they're all in the 800-1000 lines region.
Summary:
I hate Go, So Go Lost this race.
I love D, So it won or tied with C which I don't like but everyone knows is fast.
I wanted to learn Erlang, so I made sure it did better than Go.
I didn't share my SourceCode so you can't see if I was stupid or cheated.
Did I miss anything?
I hate Go, So Go Lost this race.
I love D, So it won or tied with C which I don't like but everyone knows is fast.
I wanted to learn Erlang, so I made sure it did better than Go.
I didn't share my SourceCode so you can't see if I was stupid or cheated.
Did I miss anything?
Quite a few things.
The Go version was written by a person enthusiastic and presumably competent in Go. See their perspective here where he concedes defeat on the performance front: http://blog.nella.org/mqtt-code-golf/
Adding C as a baseline is entirely appropriate.
The source code for C, Go, and D is available. The Erlang version may or may not at the discretion of its author.
Probably a more sensible hypothesis is that you like Go and saw a result that is unfavorable to it, so you had an emotional response.
The Go version was written by a person enthusiastic and presumably competent in Go. See their perspective here where he concedes defeat on the performance front: http://blog.nella.org/mqtt-code-golf/
Adding C as a baseline is entirely appropriate.
The source code for C, Go, and D is available. The Erlang version may or may not at the discretion of its author.
Probably a more sensible hypothesis is that you like Go and saw a result that is unfavorable to it, so you had an emotional response.
> As usual, Go programs compare quite favorably to C programs in terms of memory safety...
That's not saying much :)
That's not saying much :)
Source links for everything but the Erlang version are there at the end of the article.
yes, indeed you did. he did not wrote the erlang implementation but a colleague of him did.
> Go implementation was executed with go run
Why didn't they just use "go build program.go; ./program"?
If you're compiling the other programs it doesn't seem fair to use the interpreter for go.
Why didn't they just use "go build program.go; ./program"?
If you're compiling the other programs it doesn't seem fair to use the interpreter for go.
I thought "go run" did exactly that (compile then run)?
Perhaps I'm mistaken, looking at the source it does seem to do just that:
http://golang.org/src/cmd/go/run.go?ModPagespeed=noscript
I'd rather like to see Rust in here.
Me too, but I've decided to not even look at Rust until it hits 1.0.
Good, because we're certainly not ready to compete in serious benchmarks yet. :)
The results of the latency benchmark say that a larger latency is better. Is that a mistake?
No, it was measured in thousands of messages per second.
While your "benchmark" (if we should call it that way) obviously make no sense, I would suggest you guys look into how concurrency and parallelism are different concepts.
[deleted]
The key takeaway in the benchmarks seem is that Go is significantly slower than the rest, much much slower.
Apart from the regular 30-sec interval postings on HN extolling the virtues of Go, it seems to me, people should rekindle an interest, or discover a new interest in D, C or Erlang where performance is a consideration, and maybe Go where a need to feel like part of the post-Ruby crowd.
Apart from the regular 30-sec interval postings on HN extolling the virtues of Go, it seems to me, people should rekindle an interest, or discover a new interest in D, C or Erlang where performance is a consideration, and maybe Go where a need to feel like part of the post-Ruby crowd.
>Go is significantly slower than the rest, much much slower.
>Go is significantly slower than the rest, much much slower.
Further, were Go "much much slower", CloudFlare, IronIO, Google, etc would probably not be using it...
>discover a new interest in D, C or Erlang
>where performance is a consideration
Go looks to be about ~50% the performance of C++, but, from lots of peoples' comments/writings, is much nicer to write so they use it when productivity is more important than absolute performance.
Note: I'm interested in Go (as I am about lots of languages), but am not a user of it right now.
Ya, Erlang doesn't do well on the Benchmarks Game because the Benchmarks Game doesn't measure communication, which is Erlang's strong point. No-one writes algorithms like those in the Game in Erlang; you code them as a server in C or Java or anything and farm work out to them from Erlang.
> Erlang on The Game: http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t....
The Game is a terrible benchmark for Erlang. Math, computing digits if Pi, string searching, n-body problems. If that is the target domain, don't pick Erlang, or Go, pick Fortran or C.
The Game is a terrible benchmark for Erlang. Math, computing digits if Pi, string searching, n-body problems. If that is the target domain, don't pick Erlang, or Go, pick Fortran or C.
I don't understand those benchmarks, 'insert any languages' can achieve performances than 99% of companies don't even need. And for the other 1% you're probably smarter than those posts on HN.
Huh, impressive for Erlang, beating a newer, statically compiled language, while being interpreted and with dynamic types.