I'll agree on several counts (as a physicist too!):
* Rust's tooling lets you have a base-floor of what is acceptable code that is much more palatable than non-top tier Julia codes. There's a world of difference opening a random crate's code in Rust that has been linted/clippy'd/check'd by the compiler rather than a random Julia package that a coworker/colleague just fired up. This tooling is getting better and better in Julia and I'd be interested to hear which Julia projects you worked on were really hampered by TTFX and refactoring woes. Hard agree on the footguns - I want the VSCode plugin to be better at detecting dead code, it cost me weeks on a project lately.
* I'm also excited for some notion of interfaces to come to Julia. Perhaps it will be a 2.0 thing, but there's still lots of design stuff to figure out.
* We have different understandings of "productive" for different focii then - I still find it painful to do the equivalent of `rand(1:10, (20, 20))` in Rust and I can get STEM people to be productive with Julia before you can even finishing saying "borrowchecker".
At least for some scientific computing codes, but I did learn a lot of Rust along the way and extended my knowledge into (some) embedded and systems programming. Learning has been fun and the Rust people have been very friendly.
Precisely, the most natural symbiosis for a Julia + Rust combo seems to fill a hard-realtime, critical safety or embedded niche with offline simulation.
The Py03 and maturin story is something that we haven't brought to Julia yet and I'd love to have some time to work on a tighter integration story there. For example, getting a hot-reload with Revise.jl and a bindings generator that watches file changes and recompiles and the fly so that a Julia REPL gets updated with the new definitions would be so enviable.
I haven't spoken to Tim Holy to see how to string that together but many of the pieces are there.
When I wrote this, my main takeaways were numerical computations with arrays, which don't tend to use for loops in Rust, and which require grokking iterators for maximum performance and rayon-parallelism boons.
To a Julia audience, seeing this style of iterator chaining is quite a new flavor to think about composing computation, so I wanted it was with that audience in mind that I wrote this.
I have no idea why this blog is making the rounds again and I've learned a lot of Rust since, see my (badly benchmarked!) presentation at this year's Scientific Computing in Rust 2024:
I'll have to revisit this blog at some point and see what I should update, as well as a "From Rust to Julia" post for those that want to venture into that world as well.
I wanted to get too fancy and I tried
* LoopVectorization.jl - @turbo choked on the loop
* a direct llvmcall to use AVX512 pop count - I malformed the types for the instruction
* Defining the `db` as
db = [rand(Int8) for _ in 1:64, j in 1:(10^6)];
to avoid the vec of vecs structure,
and then
function my_cluster!(db, query, k)
db .= query .⊻ db
popcounts = mapreduce(count_ones, +, db, dims = 1)
results = reshape(popcounts, last(size(db)))
partialsortperm!(results, results, k)
@views results[begin:k]
end
...which I couldn't get to be faster than your version. If you use the `partialsortperm!` and reuse the same cache array, I suspect you'll get good speedups, as you won't be sorting the array every time. This is a classic `nth_element` algorithm.
The above is not the most amazing code, but I suspect the lack of indexing will make it ridiculously friendly for a GPU (Edit: Nope, it chokes on `partialsortperm!`).
I'm guessing the manual loopy approach should be just as good but I battled hard to get it somewhat competitive here in 6 lines of code
#@be my_cluster!(X2, q1, 5)
Benchmark: 3 samples with 1 evaluation
42.883 ms (17 allocs: 15.259 MiB)
45.711 ms (17 allocs: 15.259 MiB)
46.670 ms (17 allocs: 15.259 MiB)
#@be k_closest(X1, q1, 5)
Benchmark: 4 samples with 1 evaluation
27.994 ms (2 allocs: 176 bytes)
28.733 ms (2 allocs: 176 bytes)
29.000 ms (2 allocs: 176 bytes)
30.709 ms (2 allocs: 176 bytes)
I also didn't try using `FixedSizedArrays.jl` as Mose Giordano recommended in my livestream chat.
Offer still stands if you run into other bugs that aren't in our issue tracker.