Julia: Dynamism and Performance Reconciled by Design (2018)(dl.acm.org)
dl.acm.org
Julia: Dynamism and Performance Reconciled by Design (2018)
https://dl.acm.org/doi/pdf/10.1145/3276490
6 comments
Here's a broader set of benchmarks, including to R, Matlab and Fortran:
https://julialang.org/benchmarks/
AFAIK these have been optimized for the other languages, but your opinion may differ if you look at the implementations. Code for these is available here:
https://github.com/JuliaLang/Microbenchmarks
https://julialang.org/benchmarks/
AFAIK these have been optimized for the other languages, but your opinion may differ if you look at the implementations. Code for these is available here:
https://github.com/JuliaLang/Microbenchmarks
The Fortran implementation that I see on GitHub, is in no way the most optimized implementation that one could have, or would do in any serious heavy numerical computation. Yet it is amazing to see how well Fortran performs given all the lack of optimizations in the benchmark code. Also, performance highly depends on the compiler and compiler optimization flags and there is no mention of the optimization flags used for the Fortran compiler. and remember optimizations do not end with "-O3" flag in Fortran (and C). A Julia benchmark written by Julia developers showing Julia outperforms any other language does not seem to be the best way of assessing languages' performance. Did you have experts in each of the other languages to implement the respective codes?
From what I understand, the point of these benchmarks was to write relatively straightforward idiomatic code in all the laguages. The Julia devs were well aware that their knowledge of julia would make benchmark hacking straightforward, so they purposefully did not take many optimization opportunities available to them. They did the same with the Fotran and C benchmarks.
These things are all pretty subjective though when it comes to 'what is a fair benchmarking methodology'? Another valid approach would be to have experts in each language try to write as optimal an implementation as possible, but this would quickly become a bit of an arms race.
These things are all pretty subjective though when it comes to 'what is a fair benchmarking methodology'? Another valid approach would be to have experts in each language try to write as optimal an implementation as possible, but this would quickly become a bit of an arms race.
But even "relatively straightforward idiomatic code" is highly opinionated and developer dependent. For example, the array operation `a(:) = b(:) + c(:)` is very well known to be slower (and much uglier) than `a = b + c` in Fortran. Yet, many developers write in the former style and I see instances of such style usage in the Fortran code of Julia benchmark. Also, Fortran is a natively parallel concurrent vectorized language. The for-loops are automatically parallelized with `do concurrent` construct. The fact that the other languages, and perhaps Julia, lack such native capabilities, does not mean such features of Fortran should be excluded in a benchmark. Otherwise, what is really the point of a benchmark?
The point of the benchmark is to benchmark serial loops and automatic SIMD of the compilers, so no, parallelizing it doesn't make sense. In fact, Julia has native multithreaded and distributed parallelism (which can utilize MPI for the interconnects). It would be a separate test that would test native distributed parallelism, and that's not a very interesting test though since any language with these same constructs (so Julia, C++, Fortran) will all end up bandwidth limited at the same speed. This is something to give as a homework problem to students though since it's easy to demonstrate and teaches how to use the parallel constructs.
Yeah, I mean, Julia outperforming Fortran?
Come on!
Come on!
It depends on what's going on. JIT compilers have more information to optimize on so they can do surprising things. For example, FFI calls into shared libraries is generally faster with fast JIT languages.
https://github.com/dyu/ffi-overhead
This is one reason why you could see Julia outperforming Fortran in some cases where the FFI speed matters. But Fortran does have easier aliasing analysis (because you can't alias) so that helps there, but other than that most of the compiler passes are pretty much the same.
https://github.com/dyu/ffi-overhead
This is one reason why you could see Julia outperforming Fortran in some cases where the FFI speed matters. But Fortran does have easier aliasing analysis (because you can't alias) so that helps there, but other than that most of the compiler passes are pretty much the same.
That's really not that unusual.
[deleted]
I'm curious about J.
> Specificity is required to disambiguate between two or more methods which are all supertypes of the argument type. … The specificity rules are defined by the implementation and lack a formal semantics.
The lack of formal rules governing which method will be selected for a given set of arguments has made me somewhat reluctant to use Julia for large projects. Since everything in the language works off of multiple dispatch, it's a little disconcerting to have it governed by vague rules.
However, the manual [0] says that when two methods of equal specificity are present, "Julia raises a MethodError rather than arbitrarily picking a method". The manual doesn't discuss the 5 special cases in the linked PDF, though, as far as I can tell. So did the dispatch / specificity rules get formalized at some point? Or is concern about having formal rules governing dispatch unwarranted?
[0] https://docs.julialang.org/en/v1/manual/methods/
The lack of formal rules governing which method will be selected for a given set of arguments has made me somewhat reluctant to use Julia for large projects. Since everything in the language works off of multiple dispatch, it's a little disconcerting to have it governed by vague rules.
However, the manual [0] says that when two methods of equal specificity are present, "Julia raises a MethodError rather than arbitrarily picking a method". The manual doesn't discuss the 5 special cases in the linked PDF, though, as far as I can tell. So did the dispatch / specificity rules get formalized at some point? Or is concern about having formal rules governing dispatch unwarranted?
[0] https://docs.julialang.org/en/v1/manual/methods/
The subtyping system is fairly complicated, but in practice users only need to really think about situations where the LHS is a concrete type, since that's primarily what determines runtime semantics (not to say that there aren't situations where specificity really matters, but it's quite rare). That simplifies the mental model considerably and our users tend to find it relatively intuitive. For a formal discussion, the group at Northeastern has been doing great work in phrasing some of the Julia internals in more traditional PL language, including a formal subtyping model. That paper is at http://janvitek.org/pubs/oopsla18a.pdf
Love JuliaLang. But it definitely comes across as a language defined by its implementation. It is poorly specified relative to most popular languages. Appreciate the work by the paper's authors, though.
I think that's observation is fair, but I would also argue it is neither unusual (e.g. Python doesn't have one and it's probably one of the most popular language on the planet at the moment) nor a bad thing. In programming language design, you need to implement things and try out how they feel to use in practice. Programming language design is mostly about interfacing with humans, not computers. Something that has a beatiful proof on paper, but feels clumsy to use is bad programming language design. Eventually one of the efforts to make Julia an ISO standard will win and we'll be doing language design by committee and have a formal specification, but until then we'll try to build the best language we can ;).
[deleted]
One thing that does annoy me about Julia is how much they focus on performance. I do appreciate the speed but the reason why I use Julia isn't because it's fast, it's because its (in my opinion) a nicer language than R or Python/Numpy and I would like to see more effort go into that.
I do agree, but one thing I’ll say is that the performance is important because it means that you essentially need to call C code for performance. Having ll the code in a project and dependencies be in one language has some really nice advantages!
Of course we do call things like C or Python whenever there’s a good implementation of something we want, and because of that we have some excellent FFI capabilities.
But yeah, all that said, I’d use Julia even if it was slow. It’s an amazing, powerful and fun language that has taught me a lot and made me enjoy programming.
Of course we do call things like C or Python whenever there’s a good implementation of something we want, and because of that we have some excellent FFI capabilities.
But yeah, all that said, I’d use Julia even if it was slow. It’s an amazing, powerful and fun language that has taught me a lot and made me enjoy programming.
My experience has been completely the opposite. Julia has been frustrating to work with. Most of the time I’m just waiting for precompilation of the packages.
It’s unbelievable that the Julia core devs still haven’t addressed it. It is not fun by any means. Not even tolerable.
It’s unbelievable that the Julia core devs still haven’t addressed it. It is not fun by any means. Not even tolerable.
Try to use the REPL more instead of re-running your program from scratch. I just do `include("blah.jl")` then re-run my testcase.
Ah, the statefulness foot-gun. It'll solve all your problems... if your problem is having feet!
Ok, that's a little mean, there's nothing quite as convenient and simple as a REPL or notebook for ad-hoc caching, but sometimes you need the opposite and the language really needs to support that.
Ok, that's a little mean, there's nothing quite as convenient and simple as a REPL or notebook for ad-hoc caching, but sometimes you need the opposite and the language really needs to support that.
I often chain unix tools, domain-specific binary executables and python scripts in shell. Julia feels very slow for this type of workflow. A few years ago there were discussions about whether Julia could replace Python. No way just due to Julia's long startup time.
Ditto.
I can't wait for Julia to murder Python but before it can do that it needs to push all delays out of the interactive parts (to include both REPL and cold launch) and into the package management parts, where people will tolerate delays. Anaconda decided to stick a full SAT solver into the package management and it's every bit the shitshow that you'd expect, frequently taking 30 minutes to solve and failing to converge, but people tolerate it because it's in the package management.
I can't wait for Julia to murder Python but before it can do that it needs to push all delays out of the interactive parts (to include both REPL and cold launch) and into the package management parts, where people will tolerate delays. Anaconda decided to stick a full SAT solver into the package management and it's every bit the shitshow that you'd expect, frequently taking 30 minutes to solve and failing to converge, but people tolerate it because it's in the package management.
I have been using anaconda much less often due to its long resolving time... That said, I fully agree with you that Julia should cache precompiled binaries. Lacking this key functionality is a misstep from the very beginning.
That's my least favorite thing about R - it just looooves to save state unless you configure it otherwise. This is also why I've had trouble adopting Julia in the past - I've adopted the habit of constantly resetting my workspace or working from scripts to avoid exactly this issue, so my habits are somewhat Julia-hostile.
I'm giving it another go though, since it seems like it could be really useful for algorithm-adjacent research.
I'm giving it another go though, since it seems like it could be really useful for algorithm-adjacent research.
It's a bit annoying but you should only have to precompile once and it doesn't take that long
That's not true though. It's everytime I want to do anything with Julia, I need to wait upto a minute or two. Gadfly takes ages to pre-compile.
Closed your terminal session? You're outta luck. Wait 2 mins please. Launch a notebook and want to start working on something? Wait 40 seconds. Wanna check something quick in Julia REPL session? Wait 7 seconds. Let me run this Julia script, wait 55 seconds.
I feel like the entire community should stop doing what they're doing and fix this issue. I am afraid, it is difficult to fix otherwise it would have been fixed by now.
Closed your terminal session? You're outta luck. Wait 2 mins please. Launch a notebook and want to start working on something? Wait 40 seconds. Wanna check something quick in Julia REPL session? Wait 7 seconds. Let me run this Julia script, wait 55 seconds.
I feel like the entire community should stop doing what they're doing and fix this issue. I am afraid, it is difficult to fix otherwise it would have been fixed by now.
This is also roughly my experience (though I haven’t tried 1.6 yet) and it really a shame - I much prefer Julia’s syntax to R’s (my primary language) and would love to switch for statistics work.
The problem is perhaps that I largely do fairly simple statistics and plotting, with scripts that in R usually take seconds rather than minutes to run, and the disadvantages of the requirement to compile the code overcomes the compiled speed advantage.
The problem is perhaps that I largely do fairly simple statistics and plotting, with scripts that in R usually take seconds rather than minutes to run, and the disadvantages of the requirement to compile the code overcomes the compiled speed advantage.
I have the same use case as you describe and it's not unreasonable to conclude that right now Julia simply does not offer anything that R does not. It seems most Julia users are more in fields like numerical computing or scientific simulations and such, where it does offer clear advantages. But if you just want to run some of the statistical tests you learn in undergrad or want to explore some data R is just a better choice right now.
You can speed this up using a custom system image, but the usability of it kinda sucks.
https://julialang.github.io/PackageCompiler.jl/dev/examples/...
https://julialang.github.io/PackageCompiler.jl/dev/examples/...
Then they should have the package manager do it. Or the package maintainer.
That's what pushed me away in 2015. Someone said it was fixed in... 2018 I think, but I tried and it wasn't. It still isn't? Ouch.
If it comes to pushing binaries into the package manager, that's what it comes to, but waiting for a minute for the graphing library to load just isn't OK in this day and age.
If it comes to pushing binaries into the package manager, that's what it comes to, but waiting for a minute for the graphing library to load just isn't OK in this day and age.
I just tried it, and it took 13 seconds to load and plot some random numbers. It could be better but it’s an acceptable trade off for my usage. Not as good as Python in this regard, but much better than Matlab.
It's approximately halved in Julia v1.6 BTW. It's not perfect but continually getting better.
Is 13 seconds a build cache hit or miss?
It’s a precompilation hit, but it requires some compilation every time. Thinking about Julia like you might think about incremental C++ builds isn’t exactly right. A somewhat useful analogy though is that Julia is kind of like C++ template code. This is why it is so flexible, but also why compilation isn’t very cacheable. This is also why it is tricky (but possible) to do ahead of time Julia builds.
[deleted]
I work with julia every day and really don't spend much time waiting for things like compilation or precompilation.
> It’s unbelievable that the Julia core devs still haven’t addressed it. It is not fun by any means. Not even tolerable.
While I never really thought this was a big deal, the Julia devs certainly do, and I'm a little upset on their behalf that you'd say this. An amazing amount of developer time and effort has been going into reducing loading times and compiler latency lately. The currently release, 1.5.2 is leagues faster at package loading than the previous version and the upcoming 1.6 release candidate is even faster.
They recently implemented multi-threaded precomilation and have been going on a method invalidation crusade doing largely thankless work to deal with exactly this sort of whining.
> It’s unbelievable that the Julia core devs still haven’t addressed it. It is not fun by any means. Not even tolerable.
While I never really thought this was a big deal, the Julia devs certainly do, and I'm a little upset on their behalf that you'd say this. An amazing amount of developer time and effort has been going into reducing loading times and compiler latency lately. The currently release, 1.5.2 is leagues faster at package loading than the previous version and the upcoming 1.6 release candidate is even faster.
They recently implemented multi-threaded precomilation and have been going on a method invalidation crusade doing largely thankless work to deal with exactly this sort of whining.
I applaud Julia developers for their work. They just haven't focused on the UX of using Julia on a daily basis. I have been forced to use Julia as a maintainer (not author) and I feel totally exhausted, defeated and uninspired after working on Julia code.
Just being honest and voicing my opinions. If it hurts the people that worked on it for years, I am sorry.
Just being honest and voicing my opinions. If it hurts the people that worked on it for years, I am sorry.
Have you asked on https://discourse.julialang.org/ already? Maybe, hopefully, you can find help there to feel less exhausted.
-- It's not an opinion when you e.g. say that precompilation has not been addressed when it is being worked on (with much cost, I'd suppose). Same for haven't focused on the UX [snip]. In case you aren't on the julialang discourse I really encourage you to register and ask.
-- It's not an opinion when you e.g. say that precompilation has not been addressed when it is being worked on (with much cost, I'd suppose). Same for haven't focused on the UX [snip]. In case you aren't on the julialang discourse I really encourage you to register and ask.
Thanks, I haven’t reached out but mostly seek help on SO.
SO is fine. On discourse (besides asking questions like on SO) there is also space to e.g. discuss why one feels exhausted, defeated and uninspired after working with Julia. This shouldn't be! And there is certainly a way to "fix it" :)
I am not sure faster compilation is the way to go. Cutting 30 sec to 15 helps little and the acceleration of compilation can't catch up with the growth of packages in the long run. Also, multi-threaded precompilation sounds a bad idea on managed clusters where resource usage should be controlled. I wonder how much Julia devs have done on caching precompiled binaries. That, though non-trivial to implement properly, is the ultimate solution to slow compilation.
The manual says “Large modules can take several seconds to load because executing all of the statements in a module often involves compiling a large amount of code. Julia creates precompiled caches of the module to reduce this time.”
https://docs.julialang.org/en/v1/manual/modules/#Module-init...
https://docs.julialang.org/en/v1/manual/modules/#Module-init...
Just to clarify, those precompiled caches are (currently) not of binaries. At the very least, LLVM still needs to generate code after loading, and often everything up to inference still needs to be run as well.
Julia 1.6 involved a lot of work reducing invalidations, where old methods are invalidated by new definitions and thus forced to recompile. This has the obvious benefit of reducing the total amount of compilation needed, but the secondary benefit of making compiling to binaries more profitable; it's less likely that work will be wasted when someone loads another package.
Julia 1.6 involved a lot of work reducing invalidations, where old methods are invalidated by new definitions and thus forced to recompile. This has the obvious benefit of reducing the total amount of compilation needed, but the secondary benefit of making compiling to binaries more profitable; it's less likely that work will be wasted when someone loads another package.
This sounds great in theory, but in practice julia-1.5.2 still takes 10 sec to start when I use a medium-sized package including its dependencies (16 compiled files in total). If julia is loading cached files, I would expect something below 1 sec.
You can manually precompile a system image with libraries you find yourself using often, like Plots.jl. The usability of it stinks, but... you can do it, and it speeds common things up.
https://julialang.github.io/PackageCompiler.jl/dev/examples/...
https://julialang.github.io/PackageCompiler.jl/dev/examples/...
Its usability is being integrated into the package manager so that it automatically gives warnings and triggers when you need to redo the sysimage. Not in v1.6, but there's been a lot of discussions for doing it shortly after. That is something I'm excited for.
Thanks for the tip. This does look promising.
We recently acquired new hardware and plan to look into it. Invalidations can get in the way, but there are some things that can be done.
[deleted]
I don't think it's fair to say that it has not been addressed.
See this recent blog post for instance: https://julialang.org/blog/2020/08/invalidations/
See this recent blog post for instance: https://julialang.org/blog/2020/08/invalidations/
I’ve been a software engineer for many years. A lot of what I and my colleagues do is glue, sew and plumb things together.
I’ve never had to write a crazy algorithm. Nor have I had to do any graph traversal. Or anything that impacts performance.
For data analysis, pandas and numpy are plenty powerful. At no time in my career have I said “ditch all this because it’s too slow”. Perhaps others have, but I’m perfectly fine with Python. It’s a beautiful language with ability to hack into basically any part of the language. Batteries, flint, tent and a campsite included.
I’ve never had to write a crazy algorithm. Nor have I had to do any graph traversal. Or anything that impacts performance.
For data analysis, pandas and numpy are plenty powerful. At no time in my career have I said “ditch all this because it’s too slow”. Perhaps others have, but I’m perfectly fine with Python. It’s a beautiful language with ability to hack into basically any part of the language. Batteries, flint, tent and a campsite included.
CORRECTION: I am mistaken. See https://docs.python.org/3/library/ctypes.html
Python is a pretty poor glue language, in my experience.
Calling C APIs from Python is not pleasant. This is how one might import the libm "error function" from Julia:
[^1]: https://docs.python.org/3/extending/building.html
Python is a pretty poor glue language, in my experience.
Calling C APIs from Python is not pleasant. This is how one might import the libm "error function" from Julia:
const libm = "/lib64/libm.so.6"
erf(x) = @ccall libm.erf(x::Float64)::Float64
[^1]: https://docs.python.org/3/extending/building.html
Pybind11 is a dream to work with and there are python packages that implement hot reloading for C++ into a live python session aswell as autogeneration tools for it.
Python has a perfectly useful CFFI. There's no need to write an extension module.
> Calling C APIs from Python is not pleasant. [...] Python requires writing/compiling a separate dll written in C (see [^1]).
No, it doesn't, see [0] and [1].
[0]: https://docs.python.org/3/library/ctypes.html
[1]: https://cffi.readthedocs.io/en/latest/
No, it doesn't, see [0] and [1].
[0]: https://docs.python.org/3/library/ctypes.html
[1]: https://cffi.readthedocs.io/en/latest/
This the problem with anecdotal data. In the past I have spent years mostly developing and implementing new algorithms, where appropriate using graph representations. I've also spent months at a stretch on low level performance tuning.
I don't find python beautiful; it's a functional glue language and I don't hate it. It has an ok REPL and some quirky design choices. It has a decent C interop which makes it practical. What makes it useful for data analysis is the large number of packages and staggering number of person-years that have been put into them. The greater python community is great. The language is ok. Together that's pretty useful.
I don't find python beautiful; it's a functional glue language and I don't hate it. It has an ok REPL and some quirky design choices. It has a decent C interop which makes it practical. What makes it useful for data analysis is the large number of packages and staggering number of person-years that have been put into them. The greater python community is great. The language is ok. Together that's pretty useful.
I sort of agree, although there are certainly quite a few Julia users who do use it for performance.
On the other hand I much prefer Julia to Python/Numpy/Pandas regardless of speed.
somebody has to make the stuff you guys glue together.
Totally. And they can use C/C++/Rust which is what most do when performance really matters.
I don't see Julia as a general purpose langauge. I don't see high performance network stack written in Julia ever.
I see Julia as an academic scripting language that happens to be fast *
* Debatable.
I don't see Julia as a general purpose langauge. I don't see high performance network stack written in Julia ever.
I see Julia as an academic scripting language that happens to be fast *
* Debatable.
"Ever" could be a very long time, and lots of things can happen.
For example it might be possible to AoT compile parts of the code (as long as the types are fully defined) to export as a standalone library, and if that happens it would be beneficial to write the number crunching libraries for glue languages in Julia. I mean, there are already those (like DiffEq to Python and R), but if it's a binary that could have just as easily come from C it would be even better. And there are already works along this path, including for wasm deployment.
The multithreading in Julia is very fast and easy to use, and I'm sure it will become robust and safer as well as the language gets older, which can very well allow for an Akka style framework that allow for large software with both high reliability (thanks to stuff like supervision trees) and structured code (since like Julia's ecosystem design favors small libraries composing with each other, the future of large Julia codebases could become the composition of many small high level but fast actors).
There is also lots of interests in work-arounds on the garbage collector, like improved support to immutable structures (like 1.5 improvements on structs with references to mutable structures). It's possible that you might be able to more reliably write parts of the code that do not allocate (or allocate in predictable ways), so you'll be even handle those realtime code loops.
Sure this is very speculative, but look at Python today compared to when it had Julia's age. It took 8 years since Julia became public to get to this point (2 years since it got the first stable release, 1 year since it got it's multithreading model), it will surely be different when it has triple that age.
For example it might be possible to AoT compile parts of the code (as long as the types are fully defined) to export as a standalone library, and if that happens it would be beneficial to write the number crunching libraries for glue languages in Julia. I mean, there are already those (like DiffEq to Python and R), but if it's a binary that could have just as easily come from C it would be even better. And there are already works along this path, including for wasm deployment.
The multithreading in Julia is very fast and easy to use, and I'm sure it will become robust and safer as well as the language gets older, which can very well allow for an Akka style framework that allow for large software with both high reliability (thanks to stuff like supervision trees) and structured code (since like Julia's ecosystem design favors small libraries composing with each other, the future of large Julia codebases could become the composition of many small high level but fast actors).
There is also lots of interests in work-arounds on the garbage collector, like improved support to immutable structures (like 1.5 improvements on structs with references to mutable structures). It's possible that you might be able to more reliably write parts of the code that do not allocate (or allocate in predictable ways), so you'll be even handle those realtime code loops.
Sure this is very speculative, but look at Python today compared to when it had Julia's age. It took 8 years since Julia became public to get to this point (2 years since it got the first stable release, 1 year since it got it's multithreading model), it will surely be different when it has triple that age.
really? I would think the number of people who would need to use a high performance number crunching for their business logic is much higher than the number of people who need a high peroformance network stack. The network is usually pretty slow, and almost always the bottleneck is not any code that hasn't already been integrated into the OS kernel. For 99% of people, the OS network stack is fine.
That's why Julia was created though. There is a lot of scientific code that needs to be fast and is written in C or Fortran, but those languages are mediocre in terms of scientific programming.
I write crazy algorithms all the time, but I guess I have a more research oriented role. Ultimately someone has to write the crazy stuff you want to glue together, and the more productive those people are the sooner you can have those libraries available. As a user of this code I think there is a be benefit to you being able to read the source behind the algorithms without wading through esoteric FORTRAN, C, and/or CUDA.
What specifically would you like to see effort spent on? As you say, it's already a nicer language.
It would be interesting to compare it to Nim, which is also a very convenient and very performant language.
Nim has no REPL iirc, so that is a deal breaker for most interactive work IMO. There are some data science libraries coming out (maybe arraymancer), but nothing I know of in the numerical computing arena. Nim can compile to C or C++, so I assume FFI is easy, but it's still a lot of work if you aren't familiar with C. Python has all of that handled for you with certain distributions.
It does have a REPL called inim. It seems OK, but I'm not sure how it compares to python/julia if you're a REPL power user.
It's much rougher IMO. It has no autocompletion yet [0], and you cannot copy-and-paste code in it like IPython and Julia's REPL, as it does not respect the original indentation. (Tried to copy-and-paste the first example in Nim's webpage [1] and it failed because it inserted indentation at wrong places.) For the record, IPython has the `%paste` command, and Julia's REPL automatically detects when code is being pasted (this would however be less relevant, since Julia's language is not indent-based).
I am not sure if it supports plotting libraries à la matplotlib, however. If it has this capability, it might be still interesting in its present state.
[0] https://github.com/inim-repl/INim/issues/14
[1] https://nim-lang.org/
I am not sure if it supports plotting libraries à la matplotlib, however. If it has this capability, it might be still interesting in its present state.
[0] https://github.com/inim-repl/INim/issues/14
[1] https://nim-lang.org/
A plotting library: https://github.com/Vindaar/ggplotnim
Thanks for the link! Do you know if "inim" supports it (like IPython with matplotlib)? I believe this is the important point when comparing the REPL experience in Nim vs Python/Julia.
I am not an INim user. [0] covers INim as of a few months ago, though. You might need to write to a .jpg and point a web browser at it, but perhaps that could be made auto-magic (EDIT: or is already).
[0] https://www.youtube.com/watch?v=Qa_9vut4TzQ&list=PLxLdEZg8DR...
[0] https://www.youtube.com/watch?v=Qa_9vut4TzQ&list=PLxLdEZg8DR...
https://github.com/kostya/benchmarks
There are a few tests which have Julia and Nim implementations. It would be nice if the benchmarks game maintainer would include some more languages, like Nim, Crystal, and Zig.
This doesn’t have Nim, but is another interesting one. https://modelingguru.nasa.gov/docs/DOC-2783
There are a few tests which have Julia and Nim implementations. It would be nice if the benchmarks game maintainer would include some more languages, like Nim, Crystal, and Zig.
This doesn’t have Nim, but is another interesting one. https://modelingguru.nasa.gov/docs/DOC-2783
the benchmarks game maintainer is incredibly unprofessional, and looks bad for the debian team.
It would be nice if someone else would take the source code freely provided by the benchmarksgame project, include whatever languages interested them, measured the programs and published their own measurements.
I’m sure somebody has, but how would you find them? There is a network effect here. I wouldn’t mind spending the $10/month to host, but nobody would look at it. Google is basically worthless for finding anything outside of popular sites.
You would find them because they constantly did the work to promote the data in forums that were likely to have some interest.
If they did nothing to be findable they probably wouldn't be.
If they did nothing to be findable they probably wouldn't be.
So to be more precise, maybe one should say, “I wish someone would fork this project, include additional languages, and pay for better Google ranking and then pay some media company to spam HN, Medium, et al for a few months.”
If your aim is to misrepresent what was plainly written, maybe one could force those words into someone else's mouth.
I dare say that's less effort than trying to make something be successful.
I dare say that's less effort than trying to make something be successful.
[deleted]
You could probably do some tests yourself. Also as far as scientific computing goes Nim lacks a lot of libraries.
There's a tracking issue for this: https://github.com/nim-lang/needed-libraries/issues/77
I have dabbled only a bit with Julia, but I have used Python extensively, including for deep learning and for more classic machine learning applications, and my conclusion is that Julia is a superior alternative to Python in all ways except one: ecosystem.
At present, the Python ecosystem -- packages, frameworks, open-source repositories, tutorials, online discussions, communities, documentations, etc. -- is a significant advantage in favor of Python.
At present, the Python ecosystem -- packages, frameworks, open-source repositories, tutorials, online discussions, communities, documentations, etc. -- is a significant advantage in favor of Python.
To be fair that's an extremely high bar, one of the best things about Python is it's ecosystem and few other languages can match it.
> my conclusion is that Julia is a superior alternative to Python in all ways except one: ecosystem.
Fully agree, but with PyCall you can just use any Python package in Julia: https://github.com/JuliaPy/PyCall.jl
I do that regularly when I need something from Python and it works perfectly. So, Julias ecosystem is really a superset of Pythons when it comes to libraries.
Fully agree, but with PyCall you can just use any Python package in Julia: https://github.com/JuliaPy/PyCall.jl
I do that regularly when I need something from Python and it works perfectly. So, Julias ecosystem is really a superset of Pythons when it comes to libraries.
There are many areas in which Julia's ecosystem is far ahead of python's, today.
Standard ML will catch up eventually
Standard ML will catch up eventually
For me it's really interesting to read all the complaints about precompilation times etc. I'm definitely not saying the complaints are false, just that there are also some of us out there where they matter close to 0 (at current level). I use Julia from the REPL, running mostly self constructed algorithms. Only fairly basic modules are used.
My main project is a backtesting framework for algorithmic stock trading. A typical run took more than a month of runtime on my computer when the code was in optimized Numpy and now takes less than 1 minute in optimized (but likely far from optimal) Julia code. The only way to come at all close to the speed of Julia with Python would probably be something like Numba, which I figured would be more immature than Julia. Python is only usable for heavy computing on large data when the glue can be in the outermost loop so to speak.
My main project is a backtesting framework for algorithmic stock trading. A typical run took more than a month of runtime on my computer when the code was in optimized Numpy and now takes less than 1 minute in optimized (but likely far from optimal) Julia code. The only way to come at all close to the speed of Julia with Python would probably be something like Numba, which I figured would be more immature than Julia. Python is only usable for heavy computing on large data when the glue can be in the outermost loop so to speak.
I find the month to minute transformation rather shocking. Are you sure you weren't doing something algorithmicly wrong in python? Did you profile it? Like I would never leave something running for a month without putting a fair bit of time into making sure it was fast.
I don't think I did anything algorithmically wrong. If I could have done it with more of the data handling being done in c functions and just using Python as outer glue? Most likely but then that would not make the framework very extensible to other models.
Not only did I profile it, put a lot of time into optimization and learned how to use cloud computing, but I also learned a new language (Julia) and ported the whole application. :)
In Julia I can use things as data oriented design (few cache misses) to further reduce runtime. I'm not sure you get much out of such efforts in Python.
This. Having tight control over memory layout + heap allocations + vector instructions can literally decrease runtume by several orders of magnitude. Performance is so much more than having the "right algorithm". The low-level details matter so much.
Time series analysis especially goes well with Julia, as the number of dependencies is in the same order of magnitude as the number of data points.
Numpy / torch shines when the slowest part of the data processing is big matrix multiplication. In that case actually waiting for Julia compilation can slow down experimental iterations.
Numpy / torch shines when the slowest part of the data processing is big matrix multiplication. In that case actually waiting for Julia compilation can slow down experimental iterations.
That said, the Julia community has done some pretty amazing work in that subfield, last I checked they were getting performance in pure-Julia very close to the of BLAS/LAPACK, which is pretty incredible.
Julia is improving, but I wouldn't compare it to BLAS/LAPACK, cuBLAS is a much better comparision.
PyTorch uses tensor cores, supports mixed precision multiplication and bfloat16 representation.
Julia's implementation in CUDA.jl doesn't even use shared memory, which is important, as using that 10MB cache can reduce the needed main memory bandwidth significantly.
Also the new 30xx NVIDIA series supports sparse matrix multiplication, which is great if you have ReLU operations in your data processing for example. I'm not sure if PyTorch has support for that, but I'm sure it will be added, just like how mixed precision multiplication was added this year.
We're talking about an order of magnitude difference if you add all these small things together that PyTorch / cuBLAS supports. Of course Julia can call into these libraries, but then all the nice optimizations that make Julia Julia go away.
I had to think really hard of these things actually, because I love the Julia language, but the more I see ML pipelines beating the previous state-of-the-art models by 5x, the more I understand why an interpreted language with a great linear algebra library won the data processing race (I wish it would have been Ruby though).
PyTorch uses tensor cores, supports mixed precision multiplication and bfloat16 representation.
Julia's implementation in CUDA.jl doesn't even use shared memory, which is important, as using that 10MB cache can reduce the needed main memory bandwidth significantly.
Also the new 30xx NVIDIA series supports sparse matrix multiplication, which is great if you have ReLU operations in your data processing for example. I'm not sure if PyTorch has support for that, but I'm sure it will be added, just like how mixed precision multiplication was added this year.
We're talking about an order of magnitude difference if you add all these small things together that PyTorch / cuBLAS supports. Of course Julia can call into these libraries, but then all the nice optimizations that make Julia Julia go away.
I had to think really hard of these things actually, because I love the Julia language, but the more I see ML pipelines beating the previous state-of-the-art models by 5x, the more I understand why an interpreted language with a great linear algebra library won the data processing race (I wish it would have been Ruby though).
No, the parent comment is correct. Tulllio.jl has been getting up to openBLAS levels of performance
Julia also is on the cusp of support for tensor cores, mixed precision multiplication and bfloat16 codegen. New PR just merged last week
Julia also is on the cusp of support for tensor cores, mixed precision multiplication and bfloat16 codegen. New PR just merged last week
https://discourse.julialang.org/t/blas-vs-cublas-benchmark/4...
Here's an OpenBLAS vs CuBLAS performance comparision. Basically CPU is at this point outdated technology for matrix multiplication. Your second comment is more interesting, I'd be happy if Julia gets to the point when it can beat PyTorch or even JAX on training Lambda networks or Performers.
Here's an OpenBLAS vs CuBLAS performance comparision. Basically CPU is at this point outdated technology for matrix multiplication. Your second comment is more interesting, I'd be happy if Julia gets to the point when it can beat PyTorch or even JAX on training Lambda networks or Performers.
The GPU stuff is basically just a teaser into the future, it's still much more immature than that CPU stuff for Tullio.
> Basically CPU is at this point outdated technology for matrix multiplication.
This is an incredibly naive statement. There are many many circumstances where you need fast matmul on the CPU even if you have a GPU available because it takes too long to send the memory to the GPU and fetch it once the kernel runs.
If you have many chained matmuls, then yes the GPU is your friend.
That said, regardless of how useful the CPU is in practice for matrix multiplication, BLAS kernels are some of the most overengineered, most optimized pieces of code in wide use.
Tullio is incredibly flexibly and general. The fact that matrix multiplication falls out as a special case in Tullio and can outperform OpenBLAS and match MKL is absolutely stunning.
> Basically CPU is at this point outdated technology for matrix multiplication.
This is an incredibly naive statement. There are many many circumstances where you need fast matmul on the CPU even if you have a GPU available because it takes too long to send the memory to the GPU and fetch it once the kernel runs.
If you have many chained matmuls, then yes the GPU is your friend.
That said, regardless of how useful the CPU is in practice for matrix multiplication, BLAS kernels are some of the most overengineered, most optimized pieces of code in wide use.
Tullio is incredibly flexibly and general. The fact that matrix multiplication falls out as a special case in Tullio and can outperform OpenBLAS and match MKL is absolutely stunning.
Can you give concrete examples when data processing on the CPU is clearly better than on the GPU?
Before I was working with time series, and there the CPU really shined, but right now I'm classifying sick patients using DNA methylation data (about 100-300 GB / dataset). I'm using a convolutional network that already gives me state of the art results, but I want to experiment with self attention based neural networks. While I'm doing this, most of the biologists are still using logistic regression that can be computed on the CPU.
I see more and more domains (even simulations) where neural networks can outperform classical methods if you know how to apply them, and CPUs can never match the GPU performance. Also as I wrote, the slowest part of all these networks is the matrix multiplication (self attention is especially depending on it).
Before I was working with time series, and there the CPU really shined, but right now I'm classifying sick patients using DNA methylation data (about 100-300 GB / dataset). I'm using a convolutional network that already gives me state of the art results, but I want to experiment with self attention based neural networks. While I'm doing this, most of the biologists are still using logistic regression that can be computed on the CPU.
I see more and more domains (even simulations) where neural networks can outperform classical methods if you know how to apply them, and CPUs can never match the GPU performance. Also as I wrote, the slowest part of all these networks is the matrix multiplication (self attention is especially depending on it).
> Can you give concrete examples when data processing on the CPU is clearly better than on the GPU?
Pretty much anything where the matmul is important to performance, but doesn't dominate it and requires serial steps. A classic example off the top of my head would be solving a matrix differential equation[1].
Here's an example in Julia using DifferentialEquations.jl and CUDA.jl for the GPU part:
This is solving the differential equation du/dt = A * u for the cases where u and A are arrays on the GPU versus when they are arrays on the CPU. I'm doing the solving in-place to try and minimize the amount of expensive allocations.
There's not much point in using a GPU for this example, and I think that just comes down to the fact that there's a non-trivial amount of serial work that needs to be done between the matmuls.
[1] https://en.wikipedia.org/wiki/Matrix_differential_equation
Pretty much anything where the matmul is important to performance, but doesn't dominate it and requires serial steps. A classic example off the top of my head would be solving a matrix differential equation[1].
Here's an example in Julia using DifferentialEquations.jl and CUDA.jl for the GPU part:
This is solving the differential equation du/dt = A * u for the cases where u and A are arrays on the GPU versus when they are arrays on the CPU. I'm doing the solving in-place to try and minimize the amount of expensive allocations.
using DifferentialEquations, CUDA
function mysolve(u0, A, tspan)
f!(du, u, A, t) = mul!(du, A, u)
prob = ODEProblem(f!, u0, tspan, A)
solve(prob)
end
mysolve (generic function with 1 method)
julia> let n = 50, u0 = rand(Float32, n), A = randn(Float32, n, n)
# Make GPU versions of u0 and A
cuu0 = cu(u0)
cuA = cu(A)
# Do a run of the code so there's no compiler latency biasing the results
mysolve( u0, A, (0.0, 1.0))
mysolve(cuu0, cuA, (0.0, 1.0))
# Now run the compiler code and time it's execution:
@time mysolve( u0, A, (0.0, 1.0))
@time mysolve(cuu0, cuA, (0.0, 1.0))
end;
0.000314 seconds (621 allocations: 98.281 KiB)
0.007218 seconds (10.33 k allocations: 382.266 KiB)
There's not much point in using a GPU for this example, and I think that just comes down to the fact that there's a non-trivial amount of serial work that needs to be done between the matmuls.
[1] https://en.wikipedia.org/wiki/Matrix_differential_equation
I wrote big (10000x10000x10000) matrix multiplication and data processing (gigabytes of data) in the original comment, in which case the matrix size is small compared to the number of multiplications that the device has to make. Try n=10000 in your example (although I'm not sure about the differential solver part, as I don't run simulations), it's more interesting. Also differential expressions are more useful for simulation than data processing in my experience.
Of course CPU is faster for small matrices, but again, if I have hundreds of gigabytes of data that you want to process, CPU is always slower.
Of course CPU is faster for small matrices, but again, if I have hundreds of gigabytes of data that you want to process, CPU is always slower.
I understand CPU is slower, but it's not going anywhere, and matmul is just a benchmark.
It's extremely impressive that a high level (using index notation for multiple backends) Julia library can compete with hand tuned kernels. Also bodes well for Julia compiler tech generally that can easily be extended
Julia is going to get to the point of beating pytorch, just taking a but more time due to smaller team and approaching it from a more general position..so when it does it will be more flexible and ergonomic and easily extensible to new techniques, in pure Julia.
1.6 will be a big step as much of the compiler hacks underlying the current ad/gpu codegen (which just fell out accidentally of lispy design) will be replaced with proper tooling for composable compiler passes on typed IR. This will be a phase change imo.
There's already a new faster AD that's almost ready for debut based on 1.6 tech
It's extremely impressive that a high level (using index notation for multiple backends) Julia library can compete with hand tuned kernels. Also bodes well for Julia compiler tech generally that can easily be extended
Julia is going to get to the point of beating pytorch, just taking a but more time due to smaller team and approaching it from a more general position..so when it does it will be more flexible and ergonomic and easily extensible to new techniques, in pure Julia.
1.6 will be a big step as much of the compiler hacks underlying the current ad/gpu codegen (which just fell out accidentally of lispy design) will be replaced with proper tooling for composable compiler passes on typed IR. This will be a phase change imo.
There's already a new faster AD that's almost ready for debut based on 1.6 tech
That sounds amazing, thanks for the update!
I stopped using Julia because it was very frustrating that I payed thousands of dollars for a laptop with GeForce RTX 2070 card and the I couldn't make use of it, it felt like I wasted a lot of money...that was my main decision for moving to PyTorch. I know that Julia as a language should be able to beat it easily, but it felt to me that the community was prioritizing CPU over GPU for a long time. I'm happy that it's changing now.
I stopped using Julia because it was very frustrating that I payed thousands of dollars for a laptop with GeForce RTX 2070 card and the I couldn't make use of it, it felt like I wasted a lot of money...that was my main decision for moving to PyTorch. I know that Julia as a language should be able to beat it easily, but it felt to me that the community was prioritizing CPU over GPU for a long time. I'm happy that it's changing now.
Which implementation of BLAS/LAPACK? Because if it's the ref. implementation then it's not that impressive, the performance of that is very poor.
OpenBLAS and MKL.
Good point! Stuff that can be efficiently expressed as matrix operations is trivial for "Python" (eg BLAS) to handle. But I don't know of a good way to express a rolling maxima of a time series that way. And even less so if it is a streaming data feed.
What kind of architecture are you using for your backtesting framework? I have been collecting data to attempt to build something similar to that with close to around 2.3TB of tick data stretching around 20 years across 75 major pairs (indices, fx, major commods, etc) and I am always overwhelmed about design of systems to process large datasets like this.
I realize it's a bit off topic, but I just have to ask - where did you get so much ticker data?
It's been a work in progress for quite a while. Involved writing many different scrapers and other data ingestion systems that systematically downloaded everything I could get my hands on. I'm building an API to eventually sell some of the information, so I'd prefer not to say specific sources.
I'm not sure what you mean with architecture? The computers I currently run on are ordinary Macs of the latest generations. (Before Julia came and saved me I ran on Google cloud)
That is way more data than I am currently using! I'm currently a hobo using data from Yahoo finance. :) That sounds interesting! Could you contact me at kruxigt at gmail.com and tell me more?
That is way more data than I am currently using! I'm currently a hobo using data from Yahoo finance. :) That sounds interesting! Could you contact me at kruxigt at gmail.com and tell me more?
I'd love to. I'll contact you soon.
Python with Numpy and JIT, as well as Matlab, R and FORTRAN could also be included with suitable tasks. The code should be optimized in each case.