Analyzing Programming Languages Using Rosetta Code(se.inf.ethz.ch)
se.inf.ethz.ch
Analyzing Programming Languages Using Rosetta Code
http://se.inf.ethz.ch/people/nanz/research/rosettacode.html
11 comments
I know that http://www.codewars.com/ does keep track of how long it take someone to write a correct solution and could compare the different languages on that metric, does Rosetta Code have that data available or is that not measured?
Rosetta Code is simply a Wiki with a collection of tasks and a collection of implementations in various languages. So no, there is no such data. After all, you tend to not create the implementation on that site but offline and just add it to a page when you're done.
Yes, this is a problem. Some of the programming languages covered by Rosetta are "code golfing" languages which are designed to produce compact solutions for those types of problems, but are completely useless for developing anything.
Exhaustive statistical analysis and over-generalization based on inappropriate data - garbage-in garbage-out.
---
>> "… the average Go program is 18.7 times slower than the average C program. Programs in other languages are much slower than Go programs … 4.6–13.7 times slower than Go on average" page 7 <<
Elephant in the room -- What kind-of programs did the Rosetta Code program authors intend to write? Did they write their programs as-if run-time performance did not matter? Did they trade-off program performance for pretty-code / shorter programs? Did they trade-off program sophistication for program clarity?
For example, look at the gap between the Shortest C++ programs on the benchmarks game and the fastest. (I pestered a C++ program contributor to write shorter programs.)
http://benchmarksgame.alioth.debian.org/u64/code-used-time-u...
>> " …every large project is likely to include a small set of core functionalities whose quality, performance, and reliability significantly influences the whole system’s; Rosetta Code programs are indicative of such core functionalities." page 10 <<
Large projects are likely to identify that small set of core functionalities and the programmers are likely to make different trade-offs for those parts of the project, for example -- optimize performance over short code, optimize performance over programming time.
---
>> To have reliable performance measurements, the scripts repeat each execution 6 times; the timing of the first execution is discarded (to fairly accommodate bytecode languages that load virtual machines from disk: it is only in the first execution that the virtual machine is loaded from disk, with corresponding possibly significant one-time overhead; in the successive executions the virtual machine is read from cache, with only limited overhead). page 4 <<
The method does not take into account JVM dynamic optimization, and because we are not told the actual run-times of the programs there is no way to even guess whether the program has run long enough to gain the benefits.
---
>> "Which programming languages are less failure prone? To answer this question, we measure runtime failures of solutions of tasks T EXEC marked for execution that compile without errors or timeout. … Thus, a terminating program fails if it returns an exit code other than 0." page 8 <<
Why would program failure be a property of a programming language rather than an indication that the programmer did not test their program? (Or an indication that the program was written for some other implementation of the language? Or written for a workload smaller than the workload tested, given available memory?)
Even if the program did not fail, is the result correct?
---
>> "… we maintain that paradigms still significantly influence the typical “style” in which programs are written, and it is natural to associate major programming languages to a specific style based on their Rosetta Code programs. For example, even if Python offers classes and other object-oriented features, practically no solutions in Rosetta Code use them." page 11 <<
Not whether the Python programs should be considered imperative because they did not use OO features, but whether the Java and C# programs should not be considered OO because they did not go beyond the boiler-plate required by the language.
---
>> "… the average Go program is 18.7 times slower than the average C program. Programs in other languages are much slower than Go programs … 4.6–13.7 times slower than Go on average" page 7 <<
Elephant in the room -- What kind-of programs did the Rosetta Code program authors intend to write? Did they write their programs as-if run-time performance did not matter? Did they trade-off program performance for pretty-code / shorter programs? Did they trade-off program sophistication for program clarity?
For example, look at the gap between the Shortest C++ programs on the benchmarks game and the fastest. (I pestered a C++ program contributor to write shorter programs.)
http://benchmarksgame.alioth.debian.org/u64/code-used-time-u...
>> " …every large project is likely to include a small set of core functionalities whose quality, performance, and reliability significantly influences the whole system’s; Rosetta Code programs are indicative of such core functionalities." page 10 <<
Large projects are likely to identify that small set of core functionalities and the programmers are likely to make different trade-offs for those parts of the project, for example -- optimize performance over short code, optimize performance over programming time.
---
>> To have reliable performance measurements, the scripts repeat each execution 6 times; the timing of the first execution is discarded (to fairly accommodate bytecode languages that load virtual machines from disk: it is only in the first execution that the virtual machine is loaded from disk, with corresponding possibly significant one-time overhead; in the successive executions the virtual machine is read from cache, with only limited overhead). page 4 <<
The method does not take into account JVM dynamic optimization, and because we are not told the actual run-times of the programs there is no way to even guess whether the program has run long enough to gain the benefits.
---
>> "Which programming languages are less failure prone? To answer this question, we measure runtime failures of solutions of tasks T EXEC marked for execution that compile without errors or timeout. … Thus, a terminating program fails if it returns an exit code other than 0." page 8 <<
Why would program failure be a property of a programming language rather than an indication that the programmer did not test their program? (Or an indication that the program was written for some other implementation of the language? Or written for a workload smaller than the workload tested, given available memory?)
Even if the program did not fail, is the result correct?
---
>> "… we maintain that paradigms still significantly influence the typical “style” in which programs are written, and it is natural to associate major programming languages to a specific style based on their Rosetta Code programs. For example, even if Python offers classes and other object-oriented features, practically no solutions in Rosetta Code use them." page 11 <<
Not whether the Python programs should be considered imperative because they did not use OO features, but whether the Java and C# programs should not be considered OO because they did not go beyond the boiler-plate required by the language.
What should be noted, though, is that Rosetta code examples are usually not written for speed of execution, but instead in an educational manner and thus favor clearer code over faster code. At least that's how I wrote my contributions and I feel fits the aim of the project better.
Isn't that kind of the point though? Highlighting the lanuage differences proper, instead of comparing benchmarks that are optimised with non-idiomatic code?
For code length maybe. But for performance I'd find that questionable. For most things you don't need all the performance you can get from the machine. If clicking a button spends 100 cycles too many, who cares? Performance is something where you investigate when you have a problem and then optimize the part that is problematic. Sure, if performance out of the box is inacceptable and cannot even be optimized, the language should probably not used at all.
But for idiomatic code I'd find speed of execution a questionable benchmark, simply because you rarely find idiomatic code in performance-critical places. That holds for C as well as for Java. There are all sorts of rules that have to be followed to make code fast, such as no allocations in the hot path, etc. that don't usually hold for normal code.
But for idiomatic code I'd find speed of execution a questionable benchmark, simply because you rarely find idiomatic code in performance-critical places. That holds for C as well as for Java. There are all sorts of rules that have to be followed to make code fast, such as no allocations in the hot path, etc. that don't usually hold for normal code.
Or, simply: If i have language X and language Y and the most straigforward and easy-to-read version of my program is 100x faster in language X and given that both languages have the same featureset, i'll chose language X and not language Y.
Someone will always say "i can make it just as fast in language Y, i just need hideous tweaks and 10 times the LOC". But that's a moot point.
I think the interesting question is when there is a fast-and-complicated method available in both X and Y, and a slow-and-simple method available in only Y. Saying Y has to use the slow and simple method because that is "idiomatic" would be misleading, when you could be comparably fast with work comparable to doing it in X and that is the solution that would be used when performance matters.
I don't know whether this is actually a situation borne out by reality or merely a hypothetical, though.
I don't know whether this is actually a situation borne out by reality or merely a hypothetical, though.
It's still interesting to have a feel as to the point at which you have to drop out of idiomatic code.
Generally there is a complexity cost associated with performance tuning so there's value in avoiding it if possible.
Whether or not that value trumps other factors e.g. convenience, expressivity will depend on your usage but I think it's a useful data point.
Generally there is a complexity cost associated with performance tuning so there's value in avoiding it if possible.
Whether or not that value trumps other factors e.g. convenience, expressivity will depend on your usage but I think it's a useful data point.
Suppose language A offers only a very low-level way to express something, and language B offers the same thing but also a much more expressive (but less performant) way to express it. IMO it's misleading to say that language B is "slower" than language A.
Often, though, language B's low-level API won't be "language B" so much as it will be "another syntax for language A". At which point, it's often more idiomatic and simple to write in language A, than to write in language B's "version" of language A. (For just one example of this, inline assembler in C vs. actual assembler.)
The Go team must feel (deservedly) proud. Since the Rosetta Code programs are typically written to be clear, this shows that without resorting to technical tricks and knowing the intricacies of the language and its libraries Go performs better than both Java and Haskell, but of which are often lauded for their speed.
From anecdotal experience, I've written some Haskell code, and compared it to optimized versions. There is a whole lot of knowledge that goes into optimizing Haskell, since you need to know how to make things eagerly evaluate, etc. Plus functional languages make it really hard to reason about memory usage (doubly so for lazy, functional languages).
From anecdotal experience, I've written some Haskell code, and compared it to optimized versions. There is a whole lot of knowledge that goes into optimizing Haskell, since you need to know how to make things eagerly evaluate, etc. Plus functional languages make it really hard to reason about memory usage (doubly so for lazy, functional languages).
I'd say that this "benchmark" is senseless. Ruby is for sure generally not faster than Java, and C is not about 18 times faster than Go. So no reason to pat any language on the shoulder.
How is it possible that ruby is faster than Java? Are they counting jvm startup time or something? That seems rather suspicious to me.
Also weird is that Java is supposedly more consise than C#. But it turns out they're using "cloc" to measure of "lines of code", which doesn't take into account the fact that C# programs typically put opening braces on a new line.
That also struck me. Startup time is a good guess, especially considering that rosetta code snippets tend to be the kind of short programs where startup time dominates run time.
There is a potential problem using Rosetta for this kind of comparative study is that there are relatively few programmers working on the Rosetta code samples.
For some languages, the main person who does all the work is not very skilled. Perhaps that person is just doing it as a learning exercise. Or in other cases, the person just doesn't have it on their agenda to make the program as short as possible.
All else being nearly equal, languages whose Rosetta contributors fall into the above categories will "lose" to those languages for which the "Rosetta person" is an expert programmer and advocate of the language (perhaps one of the authors of the language, even!), who is working deliberately to make the code as slick as possible to promote the language, and regards Rosetta to be a competitive playing field.
For some languages, the main person who does all the work is not very skilled. Perhaps that person is just doing it as a learning exercise. Or in other cases, the person just doesn't have it on their agenda to make the program as short as possible.
All else being nearly equal, languages whose Rosetta contributors fall into the above categories will "lose" to those languages for which the "Rosetta person" is an expert programmer and advocate of the language (perhaps one of the authors of the language, even!), who is working deliberately to make the code as slick as possible to promote the language, and regards Rosetta to be a competitive playing field.
I wish they would have included Clojure or some other Lisp. I've been doing some Haskell and am actually a bit disappointed with how much less concise Haskell is compared to Clojure. I'm definitely a newb, but it feels like you have to do a lot more boilerplate/duplication especially around records.
This was partly copied by a study that Wolfram Research did on the Rosetta Code site...
http://blog.wolfram.com/2012/11/14/code-length-measured-in-1...
http://blog.wolfram.com/2012/11/14/code-length-measured-in-1...
I knew Java used a lot of memory, but that it's pretty much the worst is still kind of a shock.
Keep calm. This is not a relevant benchmark for real world scenarios.
sun used to, and now oracle does make money of machines the language is supposed to run on. now counting two and two together ... I'm not implying anything :)
Python for readability. C for speed.
One language I've been playing with lately is Lua. It's really nice how you can get it to interoperate with C code very easily. For applications that have well defined "hot loops" that need to run fast and where the rest of the code mainly exists for flow control, etc. Lua is really nice to use.
Python is also the most failure prone.
" By contrast, the scripting languages tend to be the most failure prone of the lot; Python,in particular, is significantly more failure prone than every other language"
I've asked for more info on that so I can see what needs fixing and if these non-zero-exit values appear for me.
Haskell for bragging rights.
Ruby for hipster-cred.
Hardly. Most of the new ruby development I see is in boring SaaS apps, web scrapers, and now mundane medical software at places like Hopkins. Hardly hipster stuff.
It's no longer hipster. It's mainstream.
Although what are the new hipster languages now?
Although what are the new hipster languages now?
It depends on what you're building.
E.g. for GUI-heavy apps and fast development, I favor JavaScript (or any other flavor like Coffee/Clojure/TypeScript) + React.
For execution speed and simple (or no) interface I'd choose Go, or if I could handle some platform instability I guess I would go for Rust (I still love C, but it feels like a dead weight for modern development).
E.g. for GUI-heavy apps and fast development, I favor JavaScript (or any other flavor like Coffee/Clojure/TypeScript) + React.
For execution speed and simple (or no) interface I'd choose Go, or if I could handle some platform instability I guess I would go for Rust (I still love C, but it feels like a dead weight for modern development).
I care much more about how long it takes to create/modify a program, than I care about how fast it runs or how concise it is.