Everyone should know SIMD(mitchellh.com)
mitchellh.com
Everyone should know SIMD
https://mitchellh.com/writing/everyone-should-know-simd
62 comments
This is probably one of the biggest sins in technological teaching. Sure it is crucially important to take away the fear of a topic. But you don't do so by saying it is simple, you do so by showing it is simple.
And it turns out sometimes you cannot show it is simple, because it is in fact very complex. But every complex topic is made up of smaller, simpler ones. Good teachers then manage to find a good order of those smaller parts that makes the steep hill climbable. Then you only need to convince people it is actually worth climbing.
And it turns out sometimes you cannot show it is simple, because it is in fact very complex. But every complex topic is made up of smaller, simpler ones. Good teachers then manage to find a good order of those smaller parts that makes the steep hill climbable. Then you only need to convince people it is actually worth climbing.
Simple != easy
In fact getting things to be simple can often be very hard and cumbersome.
In fact getting things to be simple can often be very hard and cumbersome.
Yep, I avoid saying the word simple almost entirely - its straight forward to get to the top of a mountain, it might still be incredibly arduous.
Simple Network Management Protocol
Simple Mail Transfer Protocol
Lightweight Directory Access Protocol
Sometimes I think the RFC editors are trolling us.
Simple Mail Transfer Protocol
Lightweight Directory Access Protocol
Sometimes I think the RFC editors are trolling us.
Well. I thought so as well about LDAP. Until one day I had to interact with the kind of technology LDAP was built to replace...
True on all counts. Not sure why you're getting downvoted.
It’s a bit weird because it is one of the better HN comments I read in the last days.
Thanks for the kind compliment : )
I try to ask myself whether things I write are (1) writing and (2) reading before writing. Sometimes that works.
I try to ask myself whether things I write are (1) writing and (2) reading before writing. Sometimes that works.
>> writing SIMD is just about as easy as a for loop
> and then the first example requires 12 lines to replace one line of scalar code.
> Be honest and say SIMD is hard but the results are worth it!
I think SIMD, and certainly that first example is way more tedious than it is hard.
What makes it tedious are
- you have to figure out how many things your hardware can do in parallel
- you have to chop up the work in packets of that size
- once you have your results, you have to ‘unchop’ them
- if there is a chance ‘chopping up’ leaves you with remaining items, you have to handle that case separately
- if the code you want to apply SIMD to uses constants, you have to create vectors containing copies of them
Neither of those is particularly hard, but each adds work, making things tedious.
> and then the first example requires 12 lines to replace one line of scalar code.
> Be honest and say SIMD is hard but the results are worth it!
I think SIMD, and certainly that first example is way more tedious than it is hard.
What makes it tedious are
- you have to figure out how many things your hardware can do in parallel
- you have to chop up the work in packets of that size
- once you have your results, you have to ‘unchop’ them
- if there is a chance ‘chopping up’ leaves you with remaining items, you have to handle that case separately
- if the code you want to apply SIMD to uses constants, you have to create vectors containing copies of them
Neither of those is particularly hard, but each adds work, making things tedious.
I hate to be the one to invoke AI in this otherwise virgin thread, but AI is absolutely terrific for doing the not-hard, tedious work. This seems like a place where AI tools could be used to help the user learn - so long as he instructs the tool for each small task and doesn't lazily just have the tool do the thinking for him.
Around 1990 I had the fortune to learn Parallel-C - a language that was designed during the Transputer hype and was essentially C extended by a few features to support easy parallel programming.
My favourite feature was
My favourite feature was
par(<start expression>; <end expression>; <loop expression>)
I believe this still lives on through XMOS. I remember (fondly) doing this on one of their chips mid-to-late 2000's doing a bit of audio processing. Think it might have even been designed by the original transputer folks.
> Okay, now I understand that those 12 lines are going to look really alien to someone not familiar with the concepts. So now let's back up and explain it step by step, mapping it directly to the shape previously mentioned.
They covered that, in a very honest and blunt manner.
They covered that, in a very honest and blunt manner.
SIMD is simple, using data-parallel operations in scalar languages is what's awkward.
Agreed, I was interested and I'm prob the target audience but things escalated too fast too quickly, very similar to the infamous "how to draw an owl" meme.
It's all relative though. The rules of playing bridge (the card game) are much harder than understanding the rules around SIMD instructions.
GF2P8AFFINE has entered the chat.
(I've both played bridge actively and written SIMD code professionally, bridge rules are way simpler. Actually playing good bridge is probably harder.)
(I've both played bridge actively and written SIMD code professionally, bridge rules are way simpler. Actually playing good bridge is probably harder.)
The lines in SIMD are much simpler though. It's like replacing one line of "I have a desire for yellow nourishment from the tropics" to "give banana" 12x
I had the same thoughts about SIMD code being too verbose when I wrote some, so a few months ago I tried writing a library that lets you write quasi-GLSL code in C++, so much more compact, with the ability to switch between SIMD width without having to rewrite anything at all:
https://github.com/gitdepierre/cppshader
Not sure it will ever be useful, but it was a fun pet project with some interesting problems to solve.
https://github.com/gitdepierre/cppshader
Not sure it will ever be useful, but it was a fun pet project with some interesting problems to solve.
Is GLSL more "approachable" than SIMD? For me personally (who doesn't have any graphics programming experience), GLSL feels way scarier than SIMD, especially when you look at the black magic that happens on shadertoys. Not saying GLSL is actually hard, but for a programmer like me SIMD might actually be more approachable
You're right, GLSL can be a bit confusing at first, especially swizzling, and the idea of writing your kernel only once then relying on the input data to drive your logic. But once you get used to it, it's very practical for writing complex stuff in just a few lines of code, it's really fun, and that's what you see a lot in Shadertoy.
I got the idea to write the library when I wanted a simplex noise function in C++, implemented with AVX2 for performance reasons (because why not). There were a lot of public HLSL/GLSL implementations that were concise and fast on GPU, some of which I had used for years for my own needs, but rewriting the whole code in plain C++ would have been a hassle, and i would have to do the same for every GPU code i came accross in the future, so building a wrapper seemed like the most efficient approach.
So in the end, the library is mostly aimed at people coming from the GPU world who want to keep most of their habits. But yeah, there is probably very few use cases for it.
I got the idea to write the library when I wanted a simplex noise function in C++, implemented with AVX2 for performance reasons (because why not). There were a lot of public HLSL/GLSL implementations that were concise and fast on GPU, some of which I had used for years for my own needs, but rewriting the whole code in plain C++ would have been a hassle, and i would have to do the same for every GPU code i came accross in the future, so building a wrapper seemed like the most efficient approach.
So in the end, the library is mostly aimed at people coming from the GPU world who want to keep most of their habits. But yeah, there is probably very few use cases for it.
What he's getting at is, the concept can be simple if you're open to understand it, and it actually is!
It's not a nitpick. The undefined acronym problem strikes yet again. It is fantastic to never use to AI write your blog posts, but please at least have it read it once. It's literally free to catch these simple writing errors and improve your writing. Vis-a-vis:
Everyone Should Know AI
AI has a reputation for being complex. I've met many very good software engineers who dismiss it as something too complex to learn or a niche meant for only highest-performance, not useful everyday. I think that's wrong. AI can be simple to understand, and common AI editorializing can speed up a blog, and almost always follows the same general shape. Once you learn the basics, editorializing with AI is just easy. And when it's not, it's usually a good sign to skip it for now. Every developer should know at least that much AI.The last few days I've been using AVX-512 to optimize matrix operations in a bioinformatics project, and it's great! The bottleneck in most applications is reading the large dataset from memory, so rather than doing it multiple times to compute multiple operations you can do everything in one pass (fused kernel) with AVX registers. 5x speedups are quite common. I've been doing it with manual intrinsics, but the wide crate also makes common operations completely trivial. Highly recommend checking it out.
https://docs.rs/wide/latest/wide/
https://docs.rs/wide/latest/wide/
I'd slightly rephrase the title to "everyone should know when SIMD didn't happen." Modern compliers are extremely good at vectorization until they suddenly aren't, an they'll often fall back to scalar code because if assumptions or a single-data dependent branch. Learning to check the compliers optimization reports is arguably more valuable.
That’s exactly what happened:
https://xcancel.com/mitchellh/status/2079672171321081908#m
https://xcancel.com/mitchellh/status/2079672171321081908#m
You know, it's always funny to read takes like "A broken compiler forcing you to write explicit SIMD instead of trusting auto-vectorization and coming out 20-30% faster is the best argument I've seen for reading your own generated assembly occasionally instead of assuming the compiler has you covered" because you can quite easily imagine an alternative one like "A broken compiler revealing that the auto-vectorization actually already accounts for 50% of total speed up of O3, and manual reimplementation and code restructuring provided only additional 20% in some scenarios is the best argument I've seen for almost never bothering with hand-crafting assembly anymore".
Is there some way to write unit tests for cases where you know vectorisation should have been applied? I guess micro benchmarks should cover the performance part. We have ArchUnit to cover code structures, it would be nice if something similar exists for generated assembly.
I've been begging for years for a a [[must_vectorize]] annotation that I can place before a loop I care about, and turn it into a compile error if the compiler can't figure it out.
Just don't do it like Rust inlining annotations:
<No annotation> – Gently suggests inlining
#[inline] - Really suggests inlining
#[inline(always)] – Really Really suggests inlining (still not guaranteed!)
https://nnethercote.github.io/perf-book/inlining.html
<No annotation> – Gently suggests inlining
#[inline] - Really suggests inlining
#[inline(always)] – Really Really suggests inlining (still not guaranteed!)
https://nnethercote.github.io/perf-book/inlining.html
this would be amazing
The gcc torture test suite has for most vect/ tests ast checks to see such regressions. LLVM missed it.
Explicit SIMD doesn’t have to mean hand-crafting assembly (and I don’t think it did in this instance).
Yeah it's strange how they brushed away that the compiler reached 77% of the hand crafted performance without even trying.
> Learning to check the compliers optimization reports is arguably more valuable.
More valuable than learning to write SIMD code? When writing SIMD code is itself the remedy to lousy autovectorisation?
If you can only identify the problem, you’re left with “well, that sucks”.
More valuable than learning to write SIMD code? When writing SIMD code is itself the remedy to lousy autovectorisation?
If you can only identify the problem, you’re left with “well, that sucks”.
> Learning to check the compliers optimization reports is arguably more valuable
Where do I start?
I want to trust the compiler, but I don't always have time to feed every little piece into compiler explorer and interpret it. Is there a higher-level workflow?
Where do I start?
I want to trust the compiler, but I don't always have time to feed every little piece into compiler explorer and interpret it. Is there a higher-level workflow?
this is amazing https://godbolt.org/, you can just paste a function or a bunch of them and instantly see what is generated. It doesn´t take being an expert to start to recognize the (auto) vectorized bits
You can write Julia code as normal and see each compiled function at the REPL. Very similar to a local Compiler Explorer.
Although as mentioned plenty of times, the largest wins tend to come from laying out your data correctly, or switching to a different algorithm. And heuristics. ;)
Although as mentioned plenty of times, the largest wins tend to come from laying out your data correctly, or switching to a different algorithm. And heuristics. ;)
To bolster the argument, even if you do not plan to write the SIMD yourself or will "just get AI to do it", it is important to know what can be fast in SIMD (and on what hardware). That allows you to design your algorithms and structure your code so that the SIMD is possible.
Internalizing things like how data dependencies matter, how expensive it is to increase the width of your vector elements (and how to avoid the need), how to turn conditions and branches into masks, or simply things like "division does not exist" becomes a lot easier when you have spent at least some time trying to use SIMD yourself.
Internalizing things like how data dependencies matter, how expensive it is to increase the width of your vector elements (and how to avoid the need), how to turn conditions and branches into masks, or simply things like "division does not exist" becomes a lot easier when you have spent at least some time trying to use SIMD yourself.
> what can be fast
I think this doesn't get talked about enough. If your input is a big run of data that is being checked/transformed in one shot, it works well. But, if you're likely to have to make a decision on several bytes of the input, SIMD will be the same or slower than the scalar method. It's not a magic "go fast" button.
I think this doesn't get talked about enough. If your input is a big run of data that is being checked/transformed in one shot, it works well. But, if you're likely to have to make a decision on several bytes of the input, SIMD will be the same or slower than the scalar method. It's not a magic "go fast" button.
That is not necessarily true. simdjson exists. But it's far from simple.
I have been told SIMD is good for data-parallel loops, like the GPU is, and that is true, but it can also be used piecemeal, unlike the GPU. Because it is just the CPU, you can read 32 unaligned bytes, scan for the index of the first space, and take a branch based on that.
I have been told SIMD is good for data-parallel loops, like the GPU is, and that is true, but it can also be used piecemeal, unlike the GPU. Because it is just the CPU, you can read 32 unaligned bytes, scan for the index of the first space, and take a branch based on that.
> That is not necessarily true. simdjson exists.
I am speaking exactly of simdjson. If you look closely, you'll see that the high performance is really only achieved by skipping over large chunks of the input data. Which is great, if your use case is operating on a subset of data. But, if the application really needs to process every piece of the entire input, there's no performance win.
Think about it, JSON structure is almost entirely made up of single bytes ({ } [ ] , : "). If you're not skipping over stuff, you're not going to get away from having to examine those bytes and branch on them. For something like `{"a":1,"b":2}`, the SIMD scan has a bunch of overhead to find the interesting parts and then you go back and practically have to reprocess every single byte.
I am speaking exactly of simdjson. If you look closely, you'll see that the high performance is really only achieved by skipping over large chunks of the input data. Which is great, if your use case is operating on a subset of data. But, if the application really needs to process every piece of the entire input, there's no performance win.
Think about it, JSON structure is almost entirely made up of single bytes ({ } [ ] , : "). If you're not skipping over stuff, you're not going to get away from having to examine those bytes and branch on them. For something like `{"a":1,"b":2}`, the SIMD scan has a bunch of overhead to find the interesting parts and then you go back and practically have to reprocess every single byte.
I think the big miss is that the former is achievable far more often than people imagine, which leads to settling for the later, aka pessimization. Reducing your allocations from millions per run to handfuls per run during initialization is effectively a "go fast" button, and is generally more reliable. It is very common for teams to spend big effort getting a 2-3x speedup on allocation-heavy code by disabling branches when a 100-1000x speedup can be had if restructuring allocations is a strategy under consideration (even before the extra 4-8x you might see if you go all the way to hand-tuned SIMD).
Here's a helpful video about leveraging SIMD to solve a concrete performance problem for the dev team that made the game The Witness by Casey Muratori: https://www.youtube.com/watch?v=Ge3aKEmZcqY
It's a great talk, I just wish there was a good focused textual version of it, as it is a very long video to recommend to others. Very worth it, but a big investment.
It's a great example of what I think of as vertical integration for performance. As you go through the talk you can understand why all these abstractions exist and why they have to be so generic. But when you have a specific use case, you can vertically integrate from the problem definition all the way down to SIMD and reap big rewards.
It's a great example of what I think of as vertical integration for performance. As you go through the talk you can understand why all these abstractions exist and why they have to be so generic. But when you have a specific use case, you can vertically integrate from the problem definition all the way down to SIMD and reap big rewards.
Be the change you want to see. Post a transcript on your own website.
Yes, I will do that when I have time one of these years. I did mean to caveat that in my post but forgot. I mainly wanted to make the vertical integration point.
Do it today or you'll end up never doing it.
Make sure to link back to the source, of course, and don't pass it off as your own words.
Make sure to link back to the source, of course, and don't pass it off as your own words.
Doesn't YouTube provide one automatically?
A transcript is not what I really want, video and writing aren't the same format. I've read transcripts of Casey's videos before.
Not in any way that's practically usable.
I like SIMD, but before super-optimizing your code with SIMD and the like, really consider your data structures and access patterns.
I've been singing Data-Oriented Design's praises, so I'll just collect all my comments here [1], but I think it's a good approach to optimization. I played around with SIMD in my old code (in Zig), but my approach to modelling datastructures was so antithetical to optimization, it was like putting high-performance racing tires on a lemon with a broken engine.
It was the root-of-all-evil-type-premature-optimization, because I wasn't measuring performance, and I wasn't thinking about where the allocations were, etc. Now, I try to model my data as if it were SQL tables, see what my potential "primary keys" could be, and build my data structures around my access patterns.
For example, I used to model trees as structs pointing to other structs on the heap:
But a tree can be represented a million ways, and can always be linearized. So now I really consider my access/insert patterns of the tree, whether it's really a tree or some other sort of graph, whether I can store it in a Vec or a Struct of Vecs, etc. Since really looking at things through their access patterns and "primary keys", my code has been much faster and simpler.
This has the added effect that a lot of your data ends up in homogeneous arrays / vecs, which means that the compiler can do its SIMD magic, the CPU can read it from your L1 cache a million times faster, etc. And then when you need to drop down into SIMD yourself, you can write some awesome branchless code.
1. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
I've been singing Data-Oriented Design's praises, so I'll just collect all my comments here [1], but I think it's a good approach to optimization. I played around with SIMD in my old code (in Zig), but my approach to modelling datastructures was so antithetical to optimization, it was like putting high-performance racing tires on a lemon with a broken engine.
It was the root-of-all-evil-type-premature-optimization, because I wasn't measuring performance, and I wasn't thinking about where the allocations were, etc. Now, I try to model my data as if it were SQL tables, see what my potential "primary keys" could be, and build my data structures around my access patterns.
For example, I used to model trees as structs pointing to other structs on the heap:
struct Tree {
tag: TreeTag,
children: Vec<&Tree>
}
But a tree can be represented a million ways, and can always be linearized. So now I really consider my access/insert patterns of the tree, whether it's really a tree or some other sort of graph, whether I can store it in a Vec or a Struct of Vecs, etc. Since really looking at things through their access patterns and "primary keys", my code has been much faster and simpler.
This has the added effect that a lot of your data ends up in homogeneous arrays / vecs, which means that the compiler can do its SIMD magic, the CPU can read it from your L1 cache a million times faster, etc. And then when you need to drop down into SIMD yourself, you can write some awesome branchless code.
1. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Yeah, data layout/cache aware layouts are really key if you really want to unlock making something that ends up in a hot loop fast with SIMD.
Also, avoiding allocations or vtable lookups or a lot of indirection in the part of the code that's actually "hot" is really important. Vectors (in C++) at least aren't necessarily the best fit either, if you end up doing anything that can call an allocation unexpectedly.
Also, avoiding allocations or vtable lookups or a lot of indirection in the part of the code that's actually "hot" is really important. Vectors (in C++) at least aren't necessarily the best fit either, if you end up doing anything that can call an allocation unexpectedly.
> Vectors (in C++) at least aren't necessarily the best fit either
I'm not sure if you use a different allocation strategy or if you're advocating allocating as much as possible up-front, but I'm curious if you have any thoughts on this:
I always end up using (Rust) vectors despite looking at a bunch of slab/arena allocation libraries. Preferably I'd know how much memory I need up front, but barring that I see three options for any allocation that needs to grow:
- Fail;
- Reallocate; or
- Put the overflow in a new allocation, keeping track of where all the "pages" are internally.
In 2, you can't use references/slices (anything with a pointer) because the potential reallocation invalidates those. In 3, it seems ideal because references can stay stable, but you wouldn't be able to have any array-like data cross the "page" barrier as the pointer jump would not be stable. (Although, am I correct in thinking the OS does something like this, and that's why pointer addresses are virtual?).
So, you can't really internally reference data in this sort of context by reference/pointer, it's preferable to use an integer index. In that case, what's the point of the allocator libraries at all? Your standard Vector would have the same reallocation/access characteristics, and you can set a reasonable initial capacity to try and avoid reallocations.
I'm not sure if you use a different allocation strategy or if you're advocating allocating as much as possible up-front, but I'm curious if you have any thoughts on this:
I always end up using (Rust) vectors despite looking at a bunch of slab/arena allocation libraries. Preferably I'd know how much memory I need up front, but barring that I see three options for any allocation that needs to grow:
- Fail;
- Reallocate; or
- Put the overflow in a new allocation, keeping track of where all the "pages" are internally.
In 2, you can't use references/slices (anything with a pointer) because the potential reallocation invalidates those. In 3, it seems ideal because references can stay stable, but you wouldn't be able to have any array-like data cross the "page" barrier as the pointer jump would not be stable. (Although, am I correct in thinking the OS does something like this, and that's why pointer addresses are virtual?).
So, you can't really internally reference data in this sort of context by reference/pointer, it's preferable to use an integer index. In that case, what's the point of the allocator libraries at all? Your standard Vector would have the same reallocation/access characteristics, and you can set a reasonable initial capacity to try and avoid reallocations.
To your virtual memory comment, the kernel can only do as it's told, but allocators can indeed tell it to shuffle pages around in virtual memory. On Linux that's `mremap` [0], and `realloc` implementations [1] use it for large enough Vecs (apparently 128kiB for glibc and musl). macOS' libmalloc will try to map new pages that extend large allocations in-place, but memcpy elsewhere if existing virtual mappings are in the way. I don't think Windows' HeapReAlloc does either, but they might reserve a larger virtual region and incrementally commit it or something to similar effect.
[0] https://man7.org/linux/man-pages/man2/mremap.2.html
[1] https://git.musl-libc.org/cgit/musl/tree/src/malloc/mallocng...
[0] https://man7.org/linux/man-pages/man2/mremap.2.html
[1] https://git.musl-libc.org/cgit/musl/tree/src/malloc/mallocng...
A fourth alternative, in 64-bit systems, is to reserve a stupidly large chunk of memory up front with `mmap()` or equivalent (`malloc()` actually should work about as well). That way you guarantee that any extension will happen in place. There’s a limit to how much you can reserve, but since that limit is much higher than what you can actually use, you can make quite a few of those reservation before you run out of address space.
It feels dirty, but when your system has overcommit you can’t reliably check that the memory you want is actually there anyway, so you might as well reap the benefits.
It feels dirty, but when your system has overcommit you can’t reliably check that the memory you want is actually there anyway, so you might as well reap the benefits.
In 32-bit systems too, but the limits are lower.
Windows is cleaner, as it lets you reserve address space and then later commit it. The Linux equivalent to reserve is probably mapping with no permissions. Overcommit can be disabled on Linux and doesn't break control flow integrity or valgrind, so I know there is a way.
Windows is cleaner, as it lets you reserve address space and then later commit it. The Linux equivalent to reserve is probably mapping with no permissions. Overcommit can be disabled on Linux and doesn't break control flow integrity or valgrind, so I know there is a way.
In Rust at least, most things have a with_capacity(n) constructor to ensure there's space for n elements (or n bytes, in the case of strings). I suppose there's no getting around the fact that if your collection has no known bounds, you'll have to do bounds checking + potential reallocation in the hot (push) path or risk having your program SIGSEGV.
https://doc.rust-lang.org/std/?search=with_capacity
https://doc.rust-lang.org/std/?search=with_capacity
You can kinda do this in C++ too with `std::vector::reserve()`. The thing is if my collection has no known bounds, well, maybe this time it requires 64GB of space? Except RAM is at a premium and I only have 16GB right now, and no swap space to speak of. What am I gonna do?
The problem with overcommit, is that there is no reliable way to know in advance how much memory your program can actually use. That system with 16GB of RAM will allow you to "allocate" 128GB of memory. Just try it, call `malloc()` or the `with_capacity()` constructor, it will return a valid pointer or container. Try to write in that giant "buffer" at the beginning, at the end, somewhere in the middle at random… it will still work. Everything works exactly as if you really had a giant 128GB buffer, that you can write to and read back from…
…Until you hit somewhere below 5M different pages, where instead of getting a new page after the page fault, you get killed by the out of memory killer. With SIGTERM if you have the relevant privilege, so you have at least a chance of exiting cleanly, but user programs just get SIGKILL. No appeal, no way to check.
Well there is a way to check, kinda: allocate a fixed amount of memory up front, hit all the pages, and if your program didn't crash, it really has the amount of memory you just gave it. Then perform all your allocations within this real buffer you really have (this means a custom allocator). Any allocation success will be real, and bounds checks will actually work. It just doesn't play well with programs whose actual memory needs are highly unpredictable: most of the time you'll use much more memory than you need, and sometimes you won't have enough, forcing you to raise the threshold.
The problem with overcommit, is that there is no reliable way to know in advance how much memory your program can actually use. That system with 16GB of RAM will allow you to "allocate" 128GB of memory. Just try it, call `malloc()` or the `with_capacity()` constructor, it will return a valid pointer or container. Try to write in that giant "buffer" at the beginning, at the end, somewhere in the middle at random… it will still work. Everything works exactly as if you really had a giant 128GB buffer, that you can write to and read back from…
…Until you hit somewhere below 5M different pages, where instead of getting a new page after the page fault, you get killed by the out of memory killer. With SIGTERM if you have the relevant privilege, so you have at least a chance of exiting cleanly, but user programs just get SIGKILL. No appeal, no way to check.
Well there is a way to check, kinda: allocate a fixed amount of memory up front, hit all the pages, and if your program didn't crash, it really has the amount of memory you just gave it. Then perform all your allocations within this real buffer you really have (this means a custom allocator). Any allocation success will be real, and bounds checks will actually work. It just doesn't play well with programs whose actual memory needs are highly unpredictable: most of the time you'll use much more memory than you need, and sometimes you won't have enough, forcing you to raise the threshold.
The main reason for virtual addresses/TLB is to prevent processes from accessing each other's memory (isolation). For optimization I'd say the page size (64 kB) is a secondary concern. It's handled in hardware (TLB) with the OS only occasionally filling up the mapping. You want to avoid that happening, but you probably should worry more using whole cache lines (64 bytes) instead, and beyond that just keep memory access local (multiple cache hierarchies) and predictable (pre-fetcher).
This is my eternal battle as a perf engineer. Performance starts with architecture and you can only squeeze so much out a hotpath with poor data layout.
The nice part is that data-oriented code almost always easily supports threading and SIMD.
The nice part is that data-oriented code almost always easily supports threading and SIMD.
On the flip side a lot of my work as a performance engineer is undoing bad abstractions made by people who read a two blog posts about SoA and decide that encapsulation is stupid
Very much agreed. Even more basic than that - memory access patterns are important. The amusing thing is that you end up writing GPU-style code even for CPU. For example - instead of an array of objects, using parquet-style object of arrays is one such trick.
There's been some good implicit/explicit discussion about SoA on this post [1]. For anyone curious, there are a few different terms that refer to basically the same thing:
- Struct of Arrays (SoA) vs Array of Structs (AoS);
- Row-major order vs column-major order; and
- Row-based vs column based / columnar (in databases)
Most code has arrays of structs (or "lists of objects", the effect is the same), and most databases are row based (same thing). But many game engines use SoA and keep heterogeneous elements together. Some databases like DuckDB do this too, this is an article about the pros and cons of columnar storage in DBs [2].
1. https://news.ycombinator.com/item?id=49012056
2. https://motherduck.com/learn/columnar-storage-guide/
- Struct of Arrays (SoA) vs Array of Structs (AoS);
- Row-major order vs column-major order; and
- Row-based vs column based / columnar (in databases)
Most code has arrays of structs (or "lists of objects", the effect is the same), and most databases are row based (same thing). But many game engines use SoA and keep heterogeneous elements together. Some databases like DuckDB do this too, this is an article about the pros and cons of columnar storage in DBs [2].
1. https://news.ycombinator.com/item?id=49012056
2. https://motherduck.com/learn/columnar-storage-guide/
This. Tables are an efficient implementation of general graphs. It’s the best one I know of (unless your graph can be specialized).
I think an even better advice is that everyone should know array programming, because you generally need that mindset for SIMD optimizations as (packed) SIMD-specific techniques are surprisingly rare. And array programming gives you a generally performant code even without SIMD because it is much easier to auto-vectorize.
I'm no fan of closed-source languages, and lord knows MATLAB has its warts. But I can't deny that it was pretty seamless to write efficient vectorised code for numerical simulations at uni. I don't have much experience with it, but my understanding is that Julia is the closest thing to a more modern and expressive language that has similar vectorisation capabilities.
R and Numpy are also pretty good at this (and Julia was inspired by both of these and Matlab).
In fact, I'm reasonably certain that R (known as S in the 70's) was the first real language designed around this concept.
In fact, I'm reasonably certain that R (known as S in the 70's) was the first real language designed around this concept.
I guess it depends on what real language means to you, but APL[1] existed before and it's fully array based.
[1]: https://en.wikipedia.org/wiki/APL_%28programming_language%29
[1]: https://en.wikipedia.org/wiki/APL_%28programming_language%29
When I writing the comment I was like, maybe APL was first? Nah, no-one will claim that. So congratulations on nerd-sniping me ;)
I learned R for statistical programming at uni, where there was less emphasis on vectorised computation, so it didn't jump to mind, but yes! I've never quite felt like numpy 'clicked' for me in the same way, though. It always felt a bit bolted on, which I suppose it is, as a library (though @ operator overloading etc makes things somewhat nicer now).
Array programming where we compare first and look for the first failure later will not help much here if runs are short because by itself it doesn't give you early termination and you may spend a lot of time on wasted comparisons.
I started learning multi-platform (x86 + ARM) SIMD last year by writing an audio synthesizer:
https://github.com/seclorum/SIMDSynth
It has been a very rewarding experience, and the synth architecture - multitimbral polyphonic - provides a great stream of data for applying SIMD principles, i.e. multiple streams going through the same process.
Has been pretty hard to debug, though. I found myself wishing I had some sort of simulator to help me understand the state of things in each pipe. I suppose I should spend some time investigating SIMD tools next time I get into this - but I fear it'll require a lot more investment. If anyone has any tips, I'm all ears ..
https://github.com/seclorum/SIMDSynth
It has been a very rewarding experience, and the synth architecture - multitimbral polyphonic - provides a great stream of data for applying SIMD principles, i.e. multiple streams going through the same process.
Has been pretty hard to debug, though. I found myself wishing I had some sort of simulator to help me understand the state of things in each pipe. I suppose I should spend some time investigating SIMD tools next time I get into this - but I fear it'll require a lot more investment. If anyone has any tips, I'm all ears ..
> Every developer should know at least that much SIMD.
> This [...] applies to any programming language. Support for SIMD instructions varies by programming language
This is a very pedantic nitpick because this article is good (& getting SIMD support across more languages would also be good), but the "every programmer should know" line feels a bit odd when neither of the 2 most popular languages natively support SIMD.
> This [...] applies to any programming language. Support for SIMD instructions varies by programming language
This is a very pedantic nitpick because this article is good (& getting SIMD support across more languages would also be good), but the "every programmer should know" line feels a bit odd when neither of the 2 most popular languages natively support SIMD.
I would venture as far as to say that the most popular languages might not be the most used by software engineers, as in, people this kind of article would be aimed at.
I'm not fully sure what this means but perhaps you're right about Python given its use by data engineers, academics, SREs (though they're mainly Go these days ... which also lacks fully stable native SIMD support). Otherwise though I'd say the rest are used almost exclusively by software engineers. Unless I'm misunderstanding you.
Still hidden behind an experimental flag but SIMD is fully functional in Go: https://pkg.go.dev/simd/archsimd
> Every developer should… most importantly, not be scared of SIMD
Seems like he should be recommending fearless_simd [1], the Rust crate by Raph Levian and the folks at Linebender :)
More seriously, if you’re looking to add SIMD to your Rust code, that’s the package to start with.
[1] https://crates.io/crates/fearless_simd
Seems like he should be recommending fearless_simd [1], the Rust crate by Raph Levian and the folks at Linebender :)
More seriously, if you’re looking to add SIMD to your Rust code, that’s the package to start with.
[1] https://crates.io/crates/fearless_simd
Tangentially for Go programming, the last time I looked at optimising some Go code with SIMD there were a few different options available, but they were either not maintained any more or had incomplete support and required first writing your function in C++ with intrinsics and generating assembly, then converting it to go assembly with a tool [1]. I never got my function to work in go despite the C++ code working fine. In short, not really a production ready option for Go. This was a year or two ago, though.
Edit, there's now an experimental official library at https://go.dev/pkg/simd/archsimd/ see https://go.dev/doc/go1.26#simd and at https://github.com/golang/go/issues/78902 so things have moved since I tried it last.
[1] https://github.com/minio/c2goasm
Edit, there's now an experimental official library at https://go.dev/pkg/simd/archsimd/ see https://go.dev/doc/go1.26#simd and at https://github.com/golang/go/issues/78902 so things have moved since I tried it last.
[1] https://github.com/minio/c2goasm
It actually works really well in the last couple Go versions with GOEXPERIMENT=simd. You do get a similar speedup (if not higher, since SIMD also eliminates the penalty for bounds checking and other things Go runtime does.
i was having a conversation with a friend recently about simd in zig (which i have recently picked up and been having a pretty good time with). i find that simd writes decently well, though there's a few weird things:
- some builtins purport to work on simd vectors but actually just unpack the vectors and do their work per-element (e.g. running `@sin()` on a `@Vector(4, f32)` will unpack the vector, run `@sin()` 4 times, and then pack it back into a vector).
- a lot of `std.math` is scalar-only (some functions support vectors, though, and i've got a pr open for one of them and plan to do more).
- i'm certainly missing some intrinsics that i get from xmmintrin.h (rcp, rsqrt, few others).
in general though i'm finding it pretty capable.
mitchell, i know you hang around some of these comments sometimes – i noticed that in ghostty you bring in some c++ libs to do the simd heavy lifting for you. any plans to port that to zig? anything missing from the language or libs that's preventing it?
- some builtins purport to work on simd vectors but actually just unpack the vectors and do their work per-element (e.g. running `@sin()` on a `@Vector(4, f32)` will unpack the vector, run `@sin()` 4 times, and then pack it back into a vector).
- a lot of `std.math` is scalar-only (some functions support vectors, though, and i've got a pr open for one of them and plan to do more).
- i'm certainly missing some intrinsics that i get from xmmintrin.h (rcp, rsqrt, few others).
in general though i'm finding it pretty capable.
mitchell, i know you hang around some of these comments sometimes – i noticed that in ghostty you bring in some c++ libs to do the simd heavy lifting for you. any plans to port that to zig? anything missing from the language or libs that's preventing it?
> mitchell, i know you hang around some of these comments sometimes
hi im here
> i noticed that in ghostty you bring in some c++ libs to do the simd heavy lifting for you. any plans to port that to zig? anything missing from the language or libs that's preventing it?
No plans to port it. For others, this is referencing highway: https://github.com/google/highway
The major limitation of Zig's vectors is that they're compile-time only. So if you're building redistributed software that compiles for a baseline CPU target, it won't be as optimized as it could be for YOUR possible machine.
Highway compiles our SIMD modules for different hardware configurations and at startup does a CPUID fingerprint to figure out which to load. That way even baseline has AVX512 etc. implementations, and we just activate the right one at runtime.
We only use Highway for our hottest hot paths that we feel benefit from that specialization.
No plans to port that (although, I spent hundreds of dollars and slop-forked it into Zig with the help of this good boy GPT and it worked great actually, but I didn't want to maintain it).
hi im here
> i noticed that in ghostty you bring in some c++ libs to do the simd heavy lifting for you. any plans to port that to zig? anything missing from the language or libs that's preventing it?
No plans to port it. For others, this is referencing highway: https://github.com/google/highway
The major limitation of Zig's vectors is that they're compile-time only. So if you're building redistributed software that compiles for a baseline CPU target, it won't be as optimized as it could be for YOUR possible machine.
Highway compiles our SIMD modules for different hardware configurations and at startup does a CPUID fingerprint to figure out which to load. That way even baseline has AVX512 etc. implementations, and we just activate the right one at runtime.
We only use Highway for our hottest hot paths that we feel benefit from that specialization.
No plans to port that (although, I spent hundreds of dollars and slop-forked it into Zig with the help of this good boy GPT and it worked great actually, but I didn't want to maintain it).
ahaaa, yeah, i don't personally do any runtime switching but i hear that as a deal-breaker from other folks.
it's interesting – i've found that zig tends to extend my vectors to the native width of the platform and then operate on them there. e.g. i had a `@Vector(2, f32)` that i was using as a demo and the generated assembly was promoting it to 256 bits and using avx2 instructions on it!
it's interesting – i've found that zig tends to extend my vectors to the native width of the platform and then operate on them there. e.g. i had a `@Vector(2, f32)` that i was using as a demo and the generated assembly was promoting it to 256 bits and using avx2 instructions on it!
> i've found that zig tends to extend my vectors to the native width of the platform and then operate on them there
oh interesting. though i suspect that isn't zig and thats llvm.
oh interesting. though i suspect that isn't zig and thats llvm.
looks like it – compiled as debug (native linux x64 backend) gives me the vector type i've asked for. release modes extend to the "native" width.
this is testing in isolation as well, could be that in the midst of other vector code it changes things. llvm definitely does a great job optimising tightly-written vector code to be even faster.
this is testing in isolation as well, could be that in the midst of other vector code it changes things. llvm definitely does a great job optimising tightly-written vector code to be even faster.
> some builtins purport to work on simd vectors but actually just unpack the vectors and do their work per-element (e.g. running `@sin()` on a `@Vector(4, f32)` will unpack the vector, run `@sin()` 4 times, and then pack it back into a vector).
this is reasonable because there isn't really a generalizable "good way" to unroll trig functions for simd. if you really care about speed youre better off implementing to the precision you care about (you might not want full precision)
this is reasonable because there isn't really a generalizable "good way" to unroll trig functions for simd. if you really care about speed youre better off implementing to the precision you care about (you might not want full precision)
The loop dependency on a poly isn't all that hard for compilers to unroll, and most correctly rounded implementations are polys. You're often paying only a couple cycles' worth of stalls.
We can see this in action. LLVM implements sin() with a correctly rounded double poly [0]. Let's ignore range reduction and throw the core into compiler explorer to be autovectorized [1]. uiCA estimates a theoretical latency for the inner loop of 15 cycles, and the code achieves 18.
I suspect most custom implementations would do worse than this.
[0] https://github.com/llvm/llvm-project/blob/165c472d65cd62eb33...
[1] https://godbolt.org/z/rGGq8aG38
We can see this in action. LLVM implements sin() with a correctly rounded double poly [0]. Let's ignore range reduction and throw the core into compiler explorer to be autovectorized [1]. uiCA estimates a theoretical latency for the inner loop of 15 cycles, and the code achieves 18.
I suspect most custom implementations would do worse than this.
[0] https://github.com/llvm/llvm-project/blob/165c472d65cd62eb33...
[1] https://godbolt.org/z/rGGq8aG38
i don't disagree – i have my own internal vector lib of approximations and whatnot for various tradeoffs of precision and speed, so i just use those. it's just that zig has a pretty strong stance of "no unexpected/obscured code execution" so it was surprising to see a vector-capable function that was just a bunch of scalar functions in a trench coat.
maybe functions that don't actually support actual vector execution just shouldn't work on vector arguments. i also wouldn't expect `@sin()` to expand in-place out to a full cephes-like sin implementation. maybe a function call.
maybe functions that don't actually support actual vector execution just shouldn't work on vector arguments. i also wouldn't expect `@sin()` to expand in-place out to a full cephes-like sin implementation. maybe a function call.
> maybe functions that don't actually support actual vector execution just shouldn't work on vector arguments.
i think that's reasonable.
i think that's reasonable.
I bet there's a better way than unpacking, running sequentially and repacking. Even if the algorithm is very branchy you save a pack and unpack.
"better" - for whom? will it cause problems if, for example you are writing scientific code?
Why isn't there? It's usually just a bunch of math and short branching that can be replaced with masks.
"More importantly, when this loop matters enough for me to care about a 5x speedup, I want the vectorization to be explicit and predictable. I don't want an unrelated code change or compiler update to quietly turn it back into a scalar loop."
Only tangentially related but this is by far the most painful part about optimizing code for JIT compilers like V8. Even changing a constant from 1 to 1.0 somewhere else can change the optimizations performed and lead to an unexpected performance decrease.
Only tangentially related but this is by far the most painful part about optimizing code for JIT compilers like V8. Even changing a constant from 1 to 1.0 somewhere else can change the optimizations performed and lead to an unexpected performance decrease.
Proving the point that the compilers aren't deterministic as some folks argue.
This is especially painful with dynamic languages, like in JavaScript's case.
However JIT also have positives hence their widespread use.
This is especially painful with dynamic languages, like in JavaScript's case.
However JIT also have positives hence their widespread use.
The Go language has long lacked official support for SIMD instructions, which means it has been at a disadvantage in terms of performance optimization. In recent years, with Go 1.26, an experimental version of the SIMD/ArchSIMD packages was introduced for AMD64 architecture. With Go 1.27, a portable version of the SIMD package was also added. Now, we can fully utilize native SIMD instructions to optimize go program performance.
- https://pkg.go.dev/simd/[email protected]
- https://pkg.go.dev/simd/[email protected]
I think too many people get dismissive of reaching out to Assembly in other languages, while in C and C++, having to reach out to Assembly to do exactly the same is seen as an advantage versus other languages.
https://github.com/kelindar/simd
https://github.com/viant/vec
However, having it officially supported is definitely much more convenient.
https://github.com/kelindar/simd
https://github.com/viant/vec
However, having it officially supported is definitely much more convenient.
The example code in this article is using Zig's portable SIMD features. Similar features are available for C/C++ (GCC/Clang extension) [0], (nightly) Rust [1] and C++26 [2].
All of these provide a similar set of features and you can use normal arithmetic operations (+, -, *, etc) for SIMD vectors. Together with templates/generics you can also write code that can deal with any vector width. These get compiled to LLVM vector types and will generally give you pretty good generated code.
This is a very good way of writing basic SIMD code and has the benefit that your code can be compiled to multiple instruction sets. I've been working on a project that can compile down to SSE2, AVX2, AVX-512 and NEON, with just a change of compiler options. Somewhat surprisingly I get the best performance by using 2x the native vector width (ie. f32x16 = 512 bits on 256 bit AVX2), which is kinda like unrolling the loop once.
There are some caveats, though. You will need to keep an eye on the generated assembly code to make sure you're on the happy path. You will inevitably need to drop down to ISA specific intrinsics every now and then (for that fast reciprocal square root with `__mm_rsqrt_ps` etc).
As an example I needed to do a gather load from an array of fp16's on AVX2, which does not do 16 bit loads. Rust's `Simd::gather_select` takes 64 bit usize as the index but AVX2 doesn't do 64 bit indices. But as long as I did all the index arithmetic in 32 bits and cast to usize at the last second, the compiler did what I wanted. But you need to kinda know what is available in the ISA to stay on the happy path. Not really an issue with arithmetic.
I'm sure that an experienced SIMD programmer can get better performance by writing intrinsics manually (say 5-20% better) but I'm already at 3-6x better than the scalar implementation I started with. And you'd have to write (and benchmark) the code for each ISA separately, meaning that you'd spend at least five times more time with it (and have 5x more code to maintain).
[0] https://gcc.gnu.org/onlinedocs/gcc-4.6.1/gcc/Vector-Extensio... [1] https://doc.rust-lang.org/nightly/std/simd/index.html [2] https://en.cppreference.com/cpp/numeric/simd
All of these provide a similar set of features and you can use normal arithmetic operations (+, -, *, etc) for SIMD vectors. Together with templates/generics you can also write code that can deal with any vector width. These get compiled to LLVM vector types and will generally give you pretty good generated code.
This is a very good way of writing basic SIMD code and has the benefit that your code can be compiled to multiple instruction sets. I've been working on a project that can compile down to SSE2, AVX2, AVX-512 and NEON, with just a change of compiler options. Somewhat surprisingly I get the best performance by using 2x the native vector width (ie. f32x16 = 512 bits on 256 bit AVX2), which is kinda like unrolling the loop once.
There are some caveats, though. You will need to keep an eye on the generated assembly code to make sure you're on the happy path. You will inevitably need to drop down to ISA specific intrinsics every now and then (for that fast reciprocal square root with `__mm_rsqrt_ps` etc).
As an example I needed to do a gather load from an array of fp16's on AVX2, which does not do 16 bit loads. Rust's `Simd::gather_select` takes 64 bit usize as the index but AVX2 doesn't do 64 bit indices. But as long as I did all the index arithmetic in 32 bits and cast to usize at the last second, the compiler did what I wanted. But you need to kinda know what is available in the ISA to stay on the happy path. Not really an issue with arithmetic.
I'm sure that an experienced SIMD programmer can get better performance by writing intrinsics manually (say 5-20% better) but I'm already at 3-6x better than the scalar implementation I started with. And you'd have to write (and benchmark) the code for each ISA separately, meaning that you'd spend at least five times more time with it (and have 5x more code to maintain).
[0] https://gcc.gnu.org/onlinedocs/gcc-4.6.1/gcc/Vector-Extensio... [1] https://doc.rust-lang.org/nightly/std/simd/index.html [2] https://en.cppreference.com/cpp/numeric/simd
FYI you linked to a really old version of the GCC documentation. Google apparently loves those old docs, so they often show up near the top of search results despite being ancient. For posterity, here's the latest version: https://gcc.gnu.org/onlinedocs/gcc-16.1.0/gcc/Vector-Extensi....
And here I am pooping out simple typescript... These articles always make me feel like I'm wasting my talents working on products that don't really have the need to leverage any understanding of what's happening under the hood at the CPU instructions level.
Writing a Gameboy emulator is a great cure for that ;)
This is an interesting article. I don't really work with low level enough languages for this to matter (unless - does this ever show up in Javascript somehow?).
I guess I don't understand the "reduce" step. It seems like you have to be careful not to "undo" all the benefit from SIMD. Sure, it can compare 8 values in parallel, but then if you have to look at each of the 8 answers in turn you're back to where you began. Is the `@reduce()` function in the example a special Vector one that tells you if all the values are true or not in one "step"?
I guess I don't understand the "reduce" step. It seems like you have to be careful not to "undo" all the benefit from SIMD. Sure, it can compare 8 values in parallel, but then if you have to look at each of the 8 answers in turn you're back to where you began. Is the `@reduce()` function in the example a special Vector one that tells you if all the values are true or not in one "step"?
> Is the `@reduce()` function in the example a special...
Yes, it is. In SIMD you can do a reduce for a commutative function in O(log N) steps. Sum, product, min, max, all, any, etc.
Although in this case it might be better if you just take the mask (vector of booleans), convert it to a bitmask and check if it is (non-) zero.
If the language provides it, you might also use .all() or .any() for a mask.
Yes, it is. In SIMD you can do a reduce for a commutative function in O(log N) steps. Sum, product, min, max, all, any, etc.
Although in this case it might be better if you just take the mask (vector of booleans), convert it to a bitmask and check if it is (non-) zero.
If the language provides it, you might also use .all() or .any() for a mask.
Yes. "`@reduce(.And, ...)` combines every boolean using `and` and returns a single boolean."
If it's true (the common case here) then you proceed to look at the next 8 bytes.
If it's false, you apply a @bitcast (turn the booleans into bits) and @ctz (find the first 0) to get the index of where it was false.
If it's true (the common case here) then you proceed to look at the next 8 bytes.
If it's false, you apply a @bitcast (turn the booleans into bits) and @ctz (find the first 0) to get the index of where it was false.
I've been using the Vector API in Java to get some massive speedups for flowfield generation. There's no guessing with that approach - if the hardware supports SIMD, you get it.
Now with Valhala finally getting merged, he can hope the end of preview releases for the Vector API is coming to an end, probably it will take at least until Java 28, though.
Here's some important accompanying advice:
* If you aren't profiling, don't bother
Otherwise, you are going to be wasting plenty of time on "optimizations" that don't actually do anything useful.
Just come up with at least a few test that represent important cases and time them. Dig in, see where the time is being spent, and focus on areas where significant time is spent, especially ones that look ripe for optimization.
Also:
* Think about laying out your data in ways that accommodate your access patterns and are cache-friendly. SIMD would typically follow from this, not be a starting point on its own.
* If you aren't profiling, don't bother
Otherwise, you are going to be wasting plenty of time on "optimizations" that don't actually do anything useful.
Just come up with at least a few test that represent important cases and time them. Dig in, see where the time is being spent, and focus on areas where significant time is spent, especially ones that look ripe for optimization.
Also:
* Think about laying out your data in ways that accommodate your access patterns and are cache-friendly. SIMD would typically follow from this, not be a starting point on its own.
It distresses me that we don’t have a language that can do a best effort parallelization of arbitrary loop like code across SIMD, multiple threads, multiple cores and GPU with a small directive.
I don’t need it to be optimal, just … handy as an option!
The last time I brought this up here, folks offered a bunch of options that don’t quite do this, and the best candidate was this 15 year old compiler project that is Intel specific!
https://ispc.github.io/
Could some programming language nerd build this?
(While you are at it give me a clear idiomatic way to pay the cost to switch from array of structs to struct of arrays)
I don’t need it to be optimal, just … handy as an option!
The last time I brought this up here, folks offered a bunch of options that don’t quite do this, and the best candidate was this 15 year old compiler project that is Intel specific!
https://ispc.github.io/
Could some programming language nerd build this?
(While you are at it give me a clear idiomatic way to pay the cost to switch from array of structs to struct of arrays)
The problem is you need both a PL nerd and a performance nerd and while that group has some overlap so these people are not as uncommon as you’d think the task is pretty hard so you need a lot of people on it, with a bunch of funding, etc. Usually it’s just cheaper to rewrite all your code by that point and so these efforts fail
It seems like such a tempting gap though. The sort of thing you’d think in 2015 would be an obvious capability of 2026 languages!
The 2015 people didn't take into account the high stress levels in 2026. We're all struggling to pay for groceries and rent.
I think I am missing the joke here. Who exactly is worse off than 11 years ago?
Pretty much everyone except the ultra wealthy.
It was bad 11 years ago. It's worse now.
It was bad 11 years ago. It's worse now.
Eggs 11 years ago cost $0.89 (hell, TWO years ago!). Today they cost $5+.
Food inflation has been moderately high but egg prices were a mix of weird short term events and umm price fixing (look up the recent case).
They are back to pre 2022 levels now! https://fred.stlouisfed.org/series/APU0000708111
They are back to pre 2022 levels now! https://fred.stlouisfed.org/series/APU0000708111
I know that it was due to price fixing, but the cartelisation of many industries and lack of any serious anti-trust enforcement (or any other kind of white-collar enforcement) are major factors impacting the economy. A 3 million dollar fine is nothing but a slap on the wrist. Even the steeper fines on dairy producers haven't stopped them from consistently raising prices at the slightest provocation.
> best effort parallelization of arbitrary loop like code across SIMD, multiple threads, multiple cores and GPU with a small directive.
I doubt GPU is included by most runtimes yet, but for the rest of that have you tried SQL?
I doubt GPU is included by most runtimes yet, but for the rest of that have you tried SQL?
Good answer, too many folks miss out how powerful SQL actually is, and with stored procedures its compilation to native code can even cached across executions.
And for an Struct-of-Arrays approach, DuckDB and other columnar databases can have nice advantages (I mentioned this in another comment [1]), including optimizations you wouldn't see in typical code (SoA or otherwise) like column-level compression [2].
The big problem with databases, in my opinion, is the horrible API friction between them and your code (not even SQL per se). It makes sense if you're calling out to a database server and transferring data, but for the small, intermediate values we see in code every day, the relational model is amazing and yet so painful to use within a given programming language.
I've been envying the C# people and their LINQ, and the Java people and their jOOQ, because I'm either making a half-assed database in my own code with structs, arrays, and hashmaps, or I'm constructing some SQL monstrosity, shoveling it out to SQLite or DuckDB through a library, and marshalling the types back and forth.
Why can't I just have everything I want all the time?
1. https://news.ycombinator.com/item?id=49016824
2. https://duckdb.org/2022/10/28/lightweight-compression
The big problem with databases, in my opinion, is the horrible API friction between them and your code (not even SQL per se). It makes sense if you're calling out to a database server and transferring data, but for the small, intermediate values we see in code every day, the relational model is amazing and yet so painful to use within a given programming language.
I've been envying the C# people and their LINQ, and the Java people and their jOOQ, because I'm either making a half-assed database in my own code with structs, arrays, and hashmaps, or I'm constructing some SQL monstrosity, shoveling it out to SQLite or DuckDB through a library, and marshalling the types back and forth.
Why can't I just have everything I want all the time?
1. https://news.ycombinator.com/item?id=49016824
2. https://duckdb.org/2022/10/28/lightweight-compression
A genuinely funny and good point!
> parallelization of arbitrary loop like code across SIMD, multiple threads, multiple cores and GPU
The reason we don’t have this is that it’s a holy grail, an unsolved problem, for quite fundamental reasons.
The other comment about SQL hints at why: SQL is largely declarative and has complex semantics built into the language, which allows for analysis and optimization that go beyond what’s possible for lower-level, general purpose languages, especially imperative ones.
For code in those languages, even just determining whether “arbitrary loop like code” is parallelizable is undecidable in general.
The challenge is that as you make a language expressive enough to describe arbitrary algorithms, you also make it progressively harder for a compiler to infer safe and useful parallel execution automatically.
Another big issue is that the various forms of parallelism are only similar at a very high level. They have fundamentally different execution models and constraints. Translating arbitrary imperative code to handle that essentially involves first inferring the intent of the code, then rewriting the code, including how data structures are organized, to fit the target architecture. This is far more than what ordinary compilers do.
There are also a lot of choices involved. Parallelism isn’t always free, so you’d need to make sure that the costs don’t outweigh the benefits - and you’d need to do that for many different decisions, like whether to use threads or not. Now you’d have a compiler building cost models to try to not make dumb choices - and without actually restarting and comparing alternatives, it’ll make mistakes.
In many ways, you’d be better off using an LLM for this, because that’s the level of understanding you need to have a hope of getting a good result.
That all said, you can do much better with more constrained languages or frameworks. SQL is the most successful example of that. Java’s streams and Rust’s Rayon only target multicore CPUs, but similar approaches could be used to do more. (Although you still potentially run into issues with optimal data shape across paradigms.) Languages like APL, J, and Futhark are all relevant.
The other family of solutions to this are the frameworks like Apache Spark, Apache Beam, and the ML frameworks like Pytorch. The latter lets you describe (tensor) computations at a high level, leaving the framework free to figure out how to implement them - much like with SQL.
The reason we don’t have this is that it’s a holy grail, an unsolved problem, for quite fundamental reasons.
The other comment about SQL hints at why: SQL is largely declarative and has complex semantics built into the language, which allows for analysis and optimization that go beyond what’s possible for lower-level, general purpose languages, especially imperative ones.
For code in those languages, even just determining whether “arbitrary loop like code” is parallelizable is undecidable in general.
The challenge is that as you make a language expressive enough to describe arbitrary algorithms, you also make it progressively harder for a compiler to infer safe and useful parallel execution automatically.
Another big issue is that the various forms of parallelism are only similar at a very high level. They have fundamentally different execution models and constraints. Translating arbitrary imperative code to handle that essentially involves first inferring the intent of the code, then rewriting the code, including how data structures are organized, to fit the target architecture. This is far more than what ordinary compilers do.
There are also a lot of choices involved. Parallelism isn’t always free, so you’d need to make sure that the costs don’t outweigh the benefits - and you’d need to do that for many different decisions, like whether to use threads or not. Now you’d have a compiler building cost models to try to not make dumb choices - and without actually restarting and comparing alternatives, it’ll make mistakes.
In many ways, you’d be better off using an LLM for this, because that’s the level of understanding you need to have a hope of getting a good result.
That all said, you can do much better with more constrained languages or frameworks. SQL is the most successful example of that. Java’s streams and Rust’s Rayon only target multicore CPUs, but similar approaches could be used to do more. (Although you still potentially run into issues with optimal data shape across paradigms.) Languages like APL, J, and Futhark are all relevant.
The other family of solutions to this are the frameworks like Apache Spark, Apache Beam, and the ML frameworks like Pytorch. The latter lets you describe (tensor) computations at a high level, leaving the framework free to figure out how to implement them - much like with SQL.
It's computer vision focused and might have been suggested previously, but I think Halide is a pretty good/mature demonstration of one way to approach this - writing the algorithm and the execution descriptions as separate passes with access to auto-optimisers and GPU runtimes.
Chapel aims to do that, however they only focus on HPC as userbase.
>It distresses me that we don’t have a language that can do a best effort parallelization of arbitrary loop like code across SIMD, multiple threads, multiple cores and GPU with a small directive.
It's called ParaSail. The next closest thing to ParaSail is ...
... literally just Rust.
Why? Because ParaSail has completely eliminated pointers, thereby preventing pointer aliasing. It can't be understated that pointer aliasing and alignment are the two biggest bottlenecks preventing autovectorization.
The people talking about better compilers, etc, just don't get it. It's not a compiler problem, it's a language semantics problem.
Pointer aliasing prevents parallelism full stop. If there is a single memory region and you perform a write to it, you have to assume that the write invalidates all data loaded from the pointers. If you make pointer aliasing illegal, then you have guaranteed that each pointer points to a distinct subset of the global memory, turning each pointer into a pointer to an isolated region. This means you have multiple regions you can write to in parallel. This is crucial, if you do not understand this you don't get parallel programming at all.
I mean think about it, this is the difference between having a four toilet bathroom with a single door or four doors.
The thing about alignment is not as easy to explain, but here is my attempt at it: If you allow the array to be misaligned at the front, then you have to run scalar code at the front. Same problem if you misalign at the end.
If you have a naked pointer and a for loop (think C), then the alignment problem alone precludes autovectorization without a complex scalar preamble. Autovectorization has to assume that the loop length could be anything, meaning it could be less than 8 elements to begin with. If the loop is always a multiple of 8 and always aligned, then the loop can be autovectorized even if the loop only does a single iteration.
Of course, after these critical blockages are gone you're still stuck with the problem of having to write branchless/non-diverging code.
It's called ParaSail. The next closest thing to ParaSail is ...
... literally just Rust.
Why? Because ParaSail has completely eliminated pointers, thereby preventing pointer aliasing. It can't be understated that pointer aliasing and alignment are the two biggest bottlenecks preventing autovectorization.
The people talking about better compilers, etc, just don't get it. It's not a compiler problem, it's a language semantics problem.
Pointer aliasing prevents parallelism full stop. If there is a single memory region and you perform a write to it, you have to assume that the write invalidates all data loaded from the pointers. If you make pointer aliasing illegal, then you have guaranteed that each pointer points to a distinct subset of the global memory, turning each pointer into a pointer to an isolated region. This means you have multiple regions you can write to in parallel. This is crucial, if you do not understand this you don't get parallel programming at all.
I mean think about it, this is the difference between having a four toilet bathroom with a single door or four doors.
The thing about alignment is not as easy to explain, but here is my attempt at it: If you allow the array to be misaligned at the front, then you have to run scalar code at the front. Same problem if you misalign at the end.
If you have a naked pointer and a for loop (think C), then the alignment problem alone precludes autovectorization without a complex scalar preamble. Autovectorization has to assume that the loop length could be anything, meaning it could be less than 8 elements to begin with. If the loop is always a multiple of 8 and always aligned, then the loop can be autovectorized even if the loop only does a single iteration.
Of course, after these critical blockages are gone you're still stuck with the problem of having to write branchless/non-diverging code.
I'd never heard of ParaSail, so I found this high-level overview [1]:
> All of the objects declared in a given scope are associated with a storage region, essentially a local heap. As an object grows, all new storage for it is allocated out of this region. As an object shrinks, the old storage can be immediately released back to this region. When a scope is exited, the entire region is reclaimed. There is no need for asynchronous garbage collection, as garbage never accumulates. Objects may grow in a highly irregular fashion without losing their locality of reference.
> Note that pointers are still used behind the scenes in the ParaSail implementation, but eliminating them from the surface syntax and semantics eliminates the complexity associated with pointers.
This approach seems to come up in many contexts, where a pointer-based address (raw pointer, reference, slice, etc.) is abstracted into a higher-level address key, usually an index integer (essentially a higher-level virtual pointer). The implementation might reallocate under the surface, or manage chunks of data through some sort of paging where the underlying pointers don't change (I was musing about this here [2]).
I guess I'm thinking out loud here, but most dynamic languages (eg. Python) don't expose the pointers or care about invalidation of the addresses, they just happily reallocate. I've barely written parallelized code, is pointer aliasing really one of the biggest roadblocks? It seems like it can be abstracted away fairly easily, even in a pointer-exposing language.
1. https://www.adacore.com/uploads/papers/parasail-pointer-free...
2. https://news.ycombinator.com/item?id=49013285
> All of the objects declared in a given scope are associated with a storage region, essentially a local heap. As an object grows, all new storage for it is allocated out of this region. As an object shrinks, the old storage can be immediately released back to this region. When a scope is exited, the entire region is reclaimed. There is no need for asynchronous garbage collection, as garbage never accumulates. Objects may grow in a highly irregular fashion without losing their locality of reference.
> Note that pointers are still used behind the scenes in the ParaSail implementation, but eliminating them from the surface syntax and semantics eliminates the complexity associated with pointers.
This approach seems to come up in many contexts, where a pointer-based address (raw pointer, reference, slice, etc.) is abstracted into a higher-level address key, usually an index integer (essentially a higher-level virtual pointer). The implementation might reallocate under the surface, or manage chunks of data through some sort of paging where the underlying pointers don't change (I was musing about this here [2]).
I guess I'm thinking out loud here, but most dynamic languages (eg. Python) don't expose the pointers or care about invalidation of the addresses, they just happily reallocate. I've barely written parallelized code, is pointer aliasing really one of the biggest roadblocks? It seems like it can be abstracted away fairly easily, even in a pointer-exposing language.
1. https://www.adacore.com/uploads/papers/parasail-pointer-free...
2. https://news.ycombinator.com/item?id=49013285
Does anyone know of a good hands-on introductory and practical tutorial on SIMD? I know that SIMD doesn't imply particular instruction set but a pattern of concurrent data transformation.
I guess what I'm looking for is something that addresses the common idioms in SIMD. For example, want to do concurrent data look up? This is how you do it in SIMD; this is how you search; this is how to prepare your data in a manner conducive to SIMD operations; these are the data types you typically find in programming languages, like __mm128; so on and so forth.
I guess what I'm looking for is something that addresses the common idioms in SIMD. For example, want to do concurrent data look up? This is how you do it in SIMD; this is how you search; this is how to prepare your data in a manner conducive to SIMD operations; these are the data types you typically find in programming languages, like __mm128; so on and so forth.
> I know that SIMD doesn't imply particular instruction set but a pattern of concurrent data transformation.
Eh. It's both. You don't have one without the other. SIMD is pretty much entirely extensions to base processor ISA; there might be a few architectures that have SIMD as a basic part of them (GPUs are one of them, taken to an extreme order), but for the most part you have to know which instruction set you're working with.
SIMD is basically "pack multiple data points into a single CPU register, then to another, and perform some operation on them as if you ran that op on all of the pairwise data points individually". Some ops are binary (arithmetic, bitwise, etc) and some are unary.
Some have "gates" whereby you can do a comparison, the boolean output of which is stored in a bit packed integer. Then you can run conditional instructions after that that only perform the instruction if the bit in that variable is set, creating "constant time" SIMD instructions with what amounts to branching. Really depends on the instruction set's capabilities.
AVX512 is far and away one of the most extensive extensions, with a ton of super niche instructions meant for enterprise number crunching. It has weird stuff, like swapping bytes, collating them, doing all sorts of weird manipulations. But the number of x86 CPUs that support that instruction set are small.
You can't emit code that has e.g. AVX512 and just run it on a CPU that doesn't have that extension. You get a CPU exception and it crashes the process (or, if this is in kernel/driver land, your machine).
So they're kind of tied together.
Anyway if you want to see a list of them, Intel's SIMD intrinsics site has always been really nice to browse IMO.
https://www.intel.com/content/www/us/en/docs/intrinsics-guid...
Eh. It's both. You don't have one without the other. SIMD is pretty much entirely extensions to base processor ISA; there might be a few architectures that have SIMD as a basic part of them (GPUs are one of them, taken to an extreme order), but for the most part you have to know which instruction set you're working with.
SIMD is basically "pack multiple data points into a single CPU register, then to another, and perform some operation on them as if you ran that op on all of the pairwise data points individually". Some ops are binary (arithmetic, bitwise, etc) and some are unary.
Some have "gates" whereby you can do a comparison, the boolean output of which is stored in a bit packed integer. Then you can run conditional instructions after that that only perform the instruction if the bit in that variable is set, creating "constant time" SIMD instructions with what amounts to branching. Really depends on the instruction set's capabilities.
AVX512 is far and away one of the most extensive extensions, with a ton of super niche instructions meant for enterprise number crunching. It has weird stuff, like swapping bytes, collating them, doing all sorts of weird manipulations. But the number of x86 CPUs that support that instruction set are small.
You can't emit code that has e.g. AVX512 and just run it on a CPU that doesn't have that extension. You get a CPU exception and it crashes the process (or, if this is in kernel/driver land, your machine).
So they're kind of tied together.
Anyway if you want to see a list of them, Intel's SIMD intrinsics site has always been really nice to browse IMO.
https://www.intel.com/content/www/us/en/docs/intrinsics-guid...
>> you'll begin to naturally decompose every for loop into these five steps
Does zig have auto vectorisation? I'm thinking that if you write the code in a vector friendly way, then the compiler can do the boiler plate for you
https://llvm.org/docs/Vectorizers.html
https://inside.java/2025/08/16/jvmls-hotspot-auto-vectorizat...
Does zig have auto vectorisation? I'm thinking that if you write the code in a vector friendly way, then the compiler can do the boiler plate for you
https://llvm.org/docs/Vectorizers.html
https://inside.java/2025/08/16/jvmls-hotspot-auto-vectorizat...
we make extensive use of simd at work, usually through highway: https://github.com/google/highway
imo this is one of the greatest libs ever written. it handles dynamic dispatching of correct simd instructions / lane widths for various hardware with just one simd loop written (handling NEON/AVX/AVX2/AVX512/extensions) with comparable performance to handwritten native intrinsics
imo this is one of the greatest libs ever written. it handles dynamic dispatching of correct simd instructions / lane widths for various hardware with just one simd loop written (handling NEON/AVX/AVX2/AVX512/extensions) with comparable performance to handwritten native intrinsics
I recommend starting with SWAR [1] before SIMD. Our registers are typically 64 bits, and one can try out SIMD patterns without taking a dependency on particular hardware.
This will only be effective if the data you’re working on is smaller than 64 bits. If you’re working with bytes, for example, you might get 8x parallelism.
[1] https://en.wikipedia.org/wiki/SWAR
This will only be effective if the data you’re working on is smaller than 64 bits. If you’re working with bytes, for example, you might get 8x parallelism.
[1] https://en.wikipedia.org/wiki/SWAR
Everyone doesn't need to know SIMD. Mechanical sympathy is an important passive perk for software architects to cut down the number of reworks down the line, but I would rate benchmarking and being able to identify bottlenecks as more important everyday skills.
I'm working on a voxel space renderer homebrew for the PlayStation. I only have so many cycles to spend on rendering before it becomes a slideshow, so I count them in my hot rendering loop and parallelize work as much stuff as I can, even across memory load stalls from main RAM.
I've worked on a basic network accessory card with a STM32 MCU that is extremely overkill for what it needs to do. We haven't bothered making any performance or memory optimizations whatsoever, writing plain C++ almost as if we were on server-class hardware because we had such egregious margins.
The first question to ask is not whether something can leverage SIMD, it's whether the performance requirements are met or not (although it's far too easy to not care when it's not your hardware that's struggling...).
I'm working on a voxel space renderer homebrew for the PlayStation. I only have so many cycles to spend on rendering before it becomes a slideshow, so I count them in my hot rendering loop and parallelize work as much stuff as I can, even across memory load stalls from main RAM.
I've worked on a basic network accessory card with a STM32 MCU that is extremely overkill for what it needs to do. We haven't bothered making any performance or memory optimizations whatsoever, writing plain C++ almost as if we were on server-class hardware because we had such egregious margins.
The first question to ask is not whether something can leverage SIMD, it's whether the performance requirements are met or not (although it's far too easy to not care when it's not your hardware that's struggling...).
> Mechanical sympathy is an important passive perk for software architects to cut down the number of reworks down the line
We might be talking on different levels but when it comes to, on an opposite end of 'Should I use SIMD'... a databases a level of mechanical sympathy at a 'base' level is still important. e.x. row-by-row updates vs batching or bad logic where a 21k entry in clause forgot about unicode rules on columns and breaks an index [0]... is still super important.
[0] - That one is real, thanks lazy bodyshop having their people use copilot and yet, we get the same billable hours, nothing is done faster, and management is too stupid to pay attention...
We might be talking on different levels but when it comes to, on an opposite end of 'Should I use SIMD'... a databases a level of mechanical sympathy at a 'base' level is still important. e.x. row-by-row updates vs batching or bad logic where a 21k entry in clause forgot about unicode rules on columns and breaks an index [0]... is still super important.
[0] - That one is real, thanks lazy bodyshop having their people use copilot and yet, we get the same billable hours, nothing is done faster, and management is too stupid to pay attention...
To do some optimization work with SIMD what you actually need to understand is the underlying CPU uarchitecture, intrinsics by the end of the day are just an API. But to also make sense of the benchmarking results or bottleneck debugging you also need to understand the underlying CPU uarchitecture and/or further devices your workload might be utilizing, e.g. storage.
> ....it's whether the performance requirements are met or not ...
This is what so many people miss when doing micro-benchmarks of language X vs Y, sure Y might win out in execution speed, however if X delivers within the performance requirements and has a lower development cost, it wins out while being slower than Y.
Naturally taken to the extreme, when it isn't our hardware is how we end up with Electron apps.
This is what so many people miss when doing micro-benchmarks of language X vs Y, sure Y might win out in execution speed, however if X delivers within the performance requirements and has a lower development cost, it wins out while being slower than Y.
Naturally taken to the extreme, when it isn't our hardware is how we end up with Electron apps.
It’s worth knowing SIMD for the purpose of knowing when it’s not worth using
Remember that bug with Intel Skylakes [0]?
When an application used AVX, it slowed down everything else on that node. It was by far not easy to debug why some applications randomly suffered perf hits on a new hardware being rolled out in Azure.
[0] https://arxiv.org/abs/1901.04982?utm_source=chatgpt.com
[0] https://arxiv.org/abs/1901.04982?utm_source=chatgpt.com
It was an early AVX-512 unit that resulted in down lock when you used more than a few instructions in short time, as well as resulted in a Linus rant.
Later CPUs (updated skylake xeons and later, AMD Zen 4 and newer) don't have the issue
Later CPUs (updated skylake xeons and later, AMD Zen 4 and newer) don't have the issue
Historically, it was a bit more complex than that, and applies to more than just AVX-512: https://gist.github.com/rygorous/32bc3ea8301dba09358fd2c64e0....
Fun! in Dwarf Fortress way :)
Thanks for the deeper dive
Thanks for the deeper dive
The Intel server CPUs Skylake Server, Cascade Lake and Cooper Lake, which had bad frequency/voltage management are now ancient history and very few of them have been used as workstation CPUs by individual users.
AMD Zen 4 and Zen 5, and also those Intel CPUs with AVX-512 support starting with Ice Lake, behave much better and there is no reason to avoid AVX-512, which has much better energy efficiency than the alternatives.
AMD Zen 4 and Zen 5, and also those Intel CPUs with AVX-512 support starting with Ice Lake, behave much better and there is no reason to avoid AVX-512, which has much better energy efficiency than the alternatives.
"Those who cannot remember the past are condemned to repeat it."
The biggest barrier I faced learning SIMD is the weird naming convention for intrinsics.
Once I got past that, it was fairly simple. mcyoung's articles were also super helpful
Once I got past that, it was fairly simple. mcyoung's articles were also super helpful
One of my favourite articles is "SIMD-friendly algorithms for substring searching" by Wojciech Muła [2]. If you were unfamiliar with SIMD and just jumped into the code, it'd be incomprehensible due to the intrinsics, but the generic algorithm description at the top is pretty simple if you take some time understand it.
It blew my mind once I understood what was happening, because it's quite clever but one of those "I could've thought of that" algorithms. There was some pretty good discussion on it last year (feat. ripgrep). [2]
1. http://0x80.pl/notesen/2016-11-28-simd-strfind.html
2. https://news.ycombinator.com/item?id=44274001
It blew my mind once I understood what was happening, because it's quite clever but one of those "I could've thought of that" algorithms. There was some pretty good discussion on it last year (feat. ripgrep). [2]
1. http://0x80.pl/notesen/2016-11-28-simd-strfind.html
2. https://news.ycombinator.com/item?id=44274001
Most of my stuff isn't CPU-bound. Most of it is in managed languages, like Bash and C# and SQL and Python and Yaml and .tsx .
It's been at least a decade for me since SIMD was more than an implementation detail handled by the runtime. And even then it was "how do I avoid preventing the runtime from vectorizing this".
It's been at least a decade for me since SIMD was more than an implementation detail handled by the runtime. And even then it was "how do I avoid preventing the runtime from vectorizing this".
I don’t know zig syntax, but wouldn’t it be possible to put this common pattern into a macro and simplify it to mostly a lambda on V?
no macros in zig, but yes you could metaprogram it. types are first class values at compile time so you could do that sort of specialization if you wanted.
There is no need to learn SIMD. Just use proper vector_size attributes. Even without the fragile auto-vectorizer, the compiler uses SIMD then. Even with -O0
I was hand-rolling NEON SIMD 15 years ago, and in many cases the compiler (clang/llvm) simply out optimized me. I kept the attempts that were better than what the compiler could already do. That was ARM NEON, quite new at the time, not SSE, and again that was 15 years ago that the compiler could already beat me much of the time. I hate the naive cult coder adage concerning premature optimization, but this might be a situation where you peruse your compiler output before you start writing code in a manner the compiler can for you. In Clang, auto-vectorization is enabled by default at optimization levels -O2 and -O3.
I like the bar and tick mark approach. I just put two percentages and then color code based on the ratio of the two for each of 5h and 7d.
Isn't the better abstraction here to use a higher level library in the style of pandas/polars that will operate as vectors, compose and feel readable and inuitive, while (almost?) maxing out SIMD?
That can easily cause you to traverse your data several times when once would be enough. Let’s say you wanted to compute “mean of array divided by max in absolute value”.
NumPy-like:
Highway:
When the actual computation is made so much faster by SIMD, memory bandwidth starts to be a significant bottleneck.
NumPy-like:
mean = np.mean(array)
maximum = np.max(np.abs(array)
return mean / maximum
Highway:
HWY_FULL(float) d;
using V = decltype(hn::Zero(d));
V sum = hn::Zero(d);
V max = hn::Zero(d);
hn::Foreach(d, values, N, hn::Zero(d), [&](auto d, auto v) HWY_ATTR {
sum = hn::Add(sum, v);
max = hn::Max(max, hn::Abs(v));
});
const float mean = hn::ReduceSum(d, sum) / N;
return mean / hn::ReduceMax(d, max);
When the actual computation is made so much faster by SIMD, memory bandwidth starts to be a significant bottleneck.
Thanks, that's insightful. I haven't thought about it that way but in retrospect it's clear. In practice I think we've all seen that the numpy style construction is quick to write and performs (and reads) much better than "dumb loops", but if you really want to optimize your code, the highway version will give you more control (and will read similar to the dumb loop with some decorations). I guess just more tools in the toolbox!
The author of this post seems to be unaware that many popular programming languages don't even support writing explicit SIMD instructions, which undermines its main thesis.
If you’re processing N things at a time and your “scalar tail” is N-1 why can’t you put in a dummy value for the last entry, run one last SIMD iteration, and discard the dummy return value?
You can, but usually the code may have problems with this. For example, storing out of bounds is often going to give you a bad time. Some platforms that are all SIMD all the time will support masked operations for this kind of thing.
There is some discussion of the possible options here: https://github.com/google/highway#strip-mining-loops
Is interesting that this is how array langs work.
I bet will be easy to turn into a lib.
I bet will be easy to turn into a lib.
I agree in spirit, but most serious cases of this class of problem have moved to accelerated kernels (e.g. GPU). Which has some commonality but is different enough that a lot of these learnings don't translate.
There might be a narrow class of problem where:
There might be a narrow class of problem where:
- SISD is the bottleneck
- Compiler won't autovectorize
- Data is small or weird enough, or the environment constrained enough that running it on an accelerator is not feasible
or when you have stronger real-time constraints than "send it to the GPU and hope it comes back fast enough" (e.g. real-time audio processing)
99% of developers should just ignore SIMD. Most projects have a lot of low hanging fruit to increase performance, and still nobody finds the time to solve them.
That doesn't mean you should default to a slower implementation for new code. If you do, you're just creating more low-hanging fruit that nobody will find time to solve.
in many cases often its faster just to switch from debug to release - compilers are good to vectorise many loops. Worth to give it a try before rewriting clean loop/code into SIMD/NEON.
In most cases you should skip SIMD also for new code. Often a non-SIMD version is required for compatibility, just stick with that.
Love Mitchell’s writing and he’s one of the few people in the industry that I truly admire.
But you need a better color scheme for your light theme my guy. Greys on greys on whites with bright pinks and light blues…it’s really hard to just read.
But you need a better color scheme for your light theme my guy. Greys on greys on whites with bright pinks and light blues…it’s really hard to just read.
I always must ask, why isn’t your compiler doing this for you? I know they often aren’t because I’ve seen speed ups from writing SIMD or using the vector functions in MKL, but this is something I really think the compilers should do for us in the simple case.
It’s really hard to make this work in general
Code should at all points be easy to reason about. Is SIMD easy to reason about? No.
[deleted]
Everyone should know SIMD, okay, but please don't introduce that kind of code into my codebase for a linear 5x improvement.
If you work on a performance-critical application, that's fine, but most of the time we use some interpreted / VM / garbage-collected slow environment that performs so much slower than just using hardware directly, and we are fine with it. Under those circumstances, it is much better to have code that every developer can understand than to introduce a 5x improvement, with code that only the person who wrote it understands.
I mean, e.g. for Go, this simplicity was one of the explicit design goals. For C / Rust / Zig, it might be a different story, but I hope you choose those languages only if they fit your use-case.
If you work on a performance-critical application, that's fine, but most of the time we use some interpreted / VM / garbage-collected slow environment that performs so much slower than just using hardware directly, and we are fine with it. Under those circumstances, it is much better to have code that every developer can understand than to introduce a 5x improvement, with code that only the person who wrote it understands.
I mean, e.g. for Go, this simplicity was one of the explicit design goals. For C / Rust / Zig, it might be a different story, but I hope you choose those languages only if they fit your use-case.
I enjoyed it. Great is the fact that examples are given.
The vast majority of developers have 0 need for learning SIMD. Why mislead them, and make them feel like to be a "real" developer they have to know it?
It seems like its useful to be aware of at the very least. Surely every developer has written a hot loop that adds or compares simple terms. Knowing that a compiler COULD in theory optimize this for the target CPU architecture is useful in many cases.
My compiler knows SIMD. However, knowing the limitations of SIMD might help avoid a calculation that can't be optimized to use it.
Your compiler can vectorise trivial things, once your code gets complicated it will no longer vectorise.
Once the code is complicated you don't need to make it more complicated lest it becomes unmaintainable.
It really doesn’t take much complexity at all for autovectorisation to fail.
Is SIMD in rust still pretty bad?
In some cases the ergonomics are worse than C, which is a feat in 2026.
Portable SIMD is (perma?) nightly.
Requires 'unsafe' everywhere.
To skip bounds-checking, you typically need to switch your writing style from loops to iterators.
No JIT, so you need multiversioning and/or target-cpu=native. Since Rust doesn't bring a compiler, you need to predict all your target architectures in advance.
Cannot inspect @code_llvm/@code_native at the function level like Julia, you need to compile the entire app.
Prioritizes Floating-Point strictness over --ffast-math. It's a genuine win for safety, but that's overkill in some domains (e.g. graphics, games, audio).
Portable SIMD is (perma?) nightly.
Requires 'unsafe' everywhere.
To skip bounds-checking, you typically need to switch your writing style from loops to iterators.
No JIT, so you need multiversioning and/or target-cpu=native. Since Rust doesn't bring a compiler, you need to predict all your target architectures in advance.
Cannot inspect @code_llvm/@code_native at the function level like Julia, you need to compile the entire app.
Prioritizes Floating-Point strictness over --ffast-math. It's a genuine win for safety, but that's overkill in some domains (e.g. graphics, games, audio).
Yes seems like too much work or hassle. I'm kinda looking at Java as a Rust replacement, and Zig as a C replacement. This might just be the ultimate combo.
I think more developers would use SIMD if there were macros handling the details.
SIMD does not pay, speaking from 20yr exp in the field in various semis.
Not really, not as SIMD remains an esoteric art from packing matrix and vector operations in endless opcodes.
I rather let the compiler auto vectorise itself, or with AI help.
I rather let the compiler auto vectorise itself, or with AI help.
Or... You know... Tell your favorite agent to "SIMD this" (:
[flagged]
"Please don't sneer, including at the rest of the community." It's reliably a marker of bad comments and worse threads.
https://news.ycombinator.com/newsguidelines.html
https://news.ycombinator.com/newsguidelines.html
No kidding. I saw no less than five "the compiler will do everything for me, I trust in the magic" comments before I gave up reading through the thread.
s/site/industry/g
I just do gcc -O3 and get SIMD without having to learn it
In the article, Mitchel mentions how this doesn’t always work. In fact, as someone who’s worked in compiler development, I can say it’s a small miracle when it does work.
Case-in-point, the example in my own post doesn't auto-vectorize with LLVM or GCC at highest optimization levels. Basically, compilers will never auto-vectorize loops with an early loop break afaik.
You would have to give the compiler some help to allow it to auto-vectorize this. Warning, untested: https://godbolt.org/z/b358bMWzG. This unrolls the loop by 16 times. The trick is using `&` instead of `and` so that there's no short-circuiting. All 16 elements are read on each iteration of the loop. This gives the compiler the freedom to replace these reads with a single 16 byte load.
> compilers will never auto-vectorize loops with an early loop break afaik.
I think this is changing. https://godbolt.org/z/ea1E7dx9v. GCC 14 won't try to vectorize this because of the break statement. But GCC 15 does vectorize it. This got a callout in the "General Improvements" section of the release notes: https://gcc.gnu.org/gcc-15/changes.html
I don't think there's any equivalent in clang/LLVM.
> compilers will never auto-vectorize loops with an early loop break afaik.
I think this is changing. https://godbolt.org/z/ea1E7dx9v. GCC 14 won't try to vectorize this because of the break statement. But GCC 15 does vectorize it. This got a callout in the "General Improvements" section of the release notes: https://gcc.gnu.org/gcc-15/changes.html
I don't think there's any equivalent in clang/LLVM.
You need to let the compiler know that there are at least 4 or 8 elements to process. This may require padding data and/or having a second loop after the main one that processes the remainder <4 or <8 elements.
You start the post with:
> There is an opportunity to use SIMD. SIMD turns those into this: > > for (8 byte chunk in bytes) { /* ... */ }
If you actually wrote that loop, there is a good chance the compiler (gcc specifically) will auto-vectorize.
In any case, the more manual SIMD optimizations I have seen require reworking the data altogether, not just processing N elements at a time. For example, instead of packing two 4-vectors into two registers to do a dot product, pack the XXXXs, YYYYs, etc. into 4 vectors and compute 4 dot products for the price of one. That not only requires having 4 vectors to process, but also thinking how exactly they are packed in registers.
I don't know why qurren is downvoted. You really should see if you can get the compiler to auto-vectorize first (possibly padding data structures and loops) before you write anything by hand.
You start the post with:
> There is an opportunity to use SIMD. SIMD turns those into this: > > for (8 byte chunk in bytes) { /* ... */ }
If you actually wrote that loop, there is a good chance the compiler (gcc specifically) will auto-vectorize.
In any case, the more manual SIMD optimizations I have seen require reworking the data altogether, not just processing N elements at a time. For example, instead of packing two 4-vectors into two registers to do a dot product, pack the XXXXs, YYYYs, etc. into 4 vectors and compute 4 dot products for the price of one. That not only requires having 4 vectors to process, but also thinking how exactly they are packed in registers.
I don't know why qurren is downvoted. You really should see if you can get the compiler to auto-vectorize first (possibly padding data structures and loops) before you write anything by hand.
> You really should see if you can get the compiler to auto-vectorize first (possibly padding data structures and loops) before you write anything by hand.
Counterpoint: https://pharr.org/matt/blog/2018/04/18/ispc-origins
> I think that the fatal flaw with the approach the compiler team was trying to make work was best diagnosed by T. Foley, who’s full of great insights about this stuff: auto-vectorization is not a programming model.
> The problem with an auto-vectorizer is that as long as vectorization can fail (and it will), then if you’re a programmer who actually cares about what code the compiler generates for your program, you must come to deeply understand the auto-vectorizer. Then, when it fails to vectorize code you want to be vectorized, you can either poke it in the right ways or change your program in the right ways so that it works for you again. This is a horrible way to program; it’s all alchemy and guesswork and you need to become deeply specialized about the nuances of a single compiler’s implementation—something you wouldn’t otherwise need to care about one bit.
> And God help you when they release a new version of the compiler with changes to the auto-vectorizer’s implementation.
> With a proper programming model, then the programmer learns the model (which is hopefully fairly clean), one or more compilers implement it, the generated code is predictable (no performance cliffs), and everyone’s happy.
Counterpoint: https://pharr.org/matt/blog/2018/04/18/ispc-origins
> I think that the fatal flaw with the approach the compiler team was trying to make work was best diagnosed by T. Foley, who’s full of great insights about this stuff: auto-vectorization is not a programming model.
> The problem with an auto-vectorizer is that as long as vectorization can fail (and it will), then if you’re a programmer who actually cares about what code the compiler generates for your program, you must come to deeply understand the auto-vectorizer. Then, when it fails to vectorize code you want to be vectorized, you can either poke it in the right ways or change your program in the right ways so that it works for you again. This is a horrible way to program; it’s all alchemy and guesswork and you need to become deeply specialized about the nuances of a single compiler’s implementation—something you wouldn’t otherwise need to care about one bit.
> And God help you when they release a new version of the compiler with changes to the auto-vectorizer’s implementation.
> With a proper programming model, then the programmer learns the model (which is hopefully fairly clean), one or more compilers implement it, the generated code is predictable (no performance cliffs), and everyone’s happy.
How is that different from any other compiler optimization?
And what does the last statement in the quote mean anyway? When is performance "predictable"; do you freeze the entire toolchain?
And what is the alternative? Handroll manual SIMD code for every possible architecture you may target?
If you're writing C++, you're already rolling on decades of compiler optimization. You return by value because it makes code more readable and safer and rely on RVO. You write functions to abstract and rely on the compiler inlining. When it doesn't work for your specific target/toolchain, you may decide to handroll stuff. The proof that you're banking on the compiler is that if you run a debug build of any non-trivial program, it runs like absolute dogshit.
And what does the last statement in the quote mean anyway? When is performance "predictable"; do you freeze the entire toolchain?
And what is the alternative? Handroll manual SIMD code for every possible architecture you may target?
If you're writing C++, you're already rolling on decades of compiler optimization. You return by value because it makes code more readable and safer and rely on RVO. You write functions to abstract and rely on the compiler inlining. When it doesn't work for your specific target/toolchain, you may decide to handroll stuff. The proof that you're banking on the compiler is that if you run a debug build of any non-trivial program, it runs like absolute dogshit.
> How is that different from any other compiler optimization?
When it can be single-handedly responsible for an 8× speedup, you might not want to rely as much on the compiler as in the case of smaller, cumulative optimisations.
> And what is the alternative? Handroll manual SIMD code for every possible architecture you may target?
Use a library like Highway? https://github.com/google/highway
When it can be single-handedly responsible for an 8× speedup, you might not want to rely as much on the compiler as in the case of smaller, cumulative optimisations.
> And what is the alternative? Handroll manual SIMD code for every possible architecture you may target?
Use a library like Highway? https://github.com/google/highway
Forgot to add: using Highway also means that you can detect and take advantage of e.g. AVX2 or AVX-512 in one binary that still works on CPUs without those instruction sets, whereas relying on autovectorisation would mean having to build with `-mavx2`, `-mavx512`, etc. which means that pretty much no one who relies on prebuilt binaries would get them.
Most scalar-to-SIMD conversion requires changing the design of data structures and algorithms to be effective. Compilers are required to exactly reproduce the specified data structures in a deterministic way for obvious reasons.
Even if compilers were clever enough to transform your data structures and algorithms for SIMD (they're not), the data structures are a contract that can't be unilaterally modified.
Even if compilers were clever enough to transform your data structures and algorithms for SIMD (they're not), the data structures are a contract that can't be unilaterally modified.
auto-vectorization is not nearly as good as you would hope it to be.
The best SIMD optimizations likely require changing your data format from AoS to SoA.
The best SIMD optimizations likely require changing your data format from AoS to SoA.
> The best SIMD optimizations likely require changing your data format from AoS to SoA.
We do have gather load instructions in SIMD instruction sets these days (AVX2 and newer), so AoS vs SoA is not nearly as important as it was once.
Scatter stores are also available but only in newer CPUs.
We do have gather load instructions in SIMD instruction sets these days (AVX2 and newer), so AoS vs SoA is not nearly as important as it was once.
Scatter stores are also available but only in newer CPUs.
It remains extremely important. Gather loads are much more expensive than sequential loads, for obvious reasons.
While C++ may be reaching levels of Algol 68, PL/I complexity, with C++26 reflection you can do automatically.
See https://github.com/cern-nextgen/reflmempp
See https://github.com/cern-nextgen/reflmempp
That’s a research project. So no.
And -march=native or at least -march=x86-64-v3 or similar, alternatively identifying relevant functions and manually invoking FMV and uarch specialization via target_clones. Plus non-integer code can generally not be autovectorized in normal-math mode since FP is non-commutative.
The one feature in Jonathan Blow's Jai language I really envy is a a single keyword to switch AoS to SoA and visa-versa at comptime
Didn't he drop this feature years ago?
Either this or you have to do special tricks like pairwise tree reductions and hand-unroll certain portions of loops.
What are AoS and SoA?
Array of Structs and Struct of Arrays https://en.wikipedia.org/wiki/AoS_and_SoA
A good introduction to SoA (for anyone curious) are the two most famous Data-Oriented Design talks by Mike Acton (game engine dev) [1] and Andrew Kelley (Zig lead dev) [2] respectively.
I read a book book about DoD [3] really which confused me at first with all its talk about database table design (in a book about a high-performance C++ game engine?), but when it finally clicked it was amazing. The point is that you want to think hard about your access patterns and what could constitute good "primary keys", then model it accordingly. SoA ends up being useful a lot of the time, because having your data in homogeneous arrays/vectors is great for cache locality and branch elimination. Even without SIMD you can get huge speedups from that, but that's also where your compiler (or you as a programmer) can get incredible SIMD gains.
SoA is not a silver bullet as it may not align well with your access patterns, but it can great to add to your toolkit.
---
Mike Acton: Data-Oriented Design and C++: https://www.youtube.com/watch?v=rX0ItVEVjHc
Andrew Kelley: A Practical Guide to Applying Data Oriented Design: https://www.youtube.com/watch?v=IroPQ150F6c
Richard Fabian: Data-Oriented Design: https://www.dataorienteddesign.com/dodbook/
I read a book book about DoD [3] really which confused me at first with all its talk about database table design (in a book about a high-performance C++ game engine?), but when it finally clicked it was amazing. The point is that you want to think hard about your access patterns and what could constitute good "primary keys", then model it accordingly. SoA ends up being useful a lot of the time, because having your data in homogeneous arrays/vectors is great for cache locality and branch elimination. Even without SIMD you can get huge speedups from that, but that's also where your compiler (or you as a programmer) can get incredible SIMD gains.
SoA is not a silver bullet as it may not align well with your access patterns, but it can great to add to your toolkit.
---
Mike Acton: Data-Oriented Design and C++: https://www.youtube.com/watch?v=rX0ItVEVjHc
Andrew Kelley: A Practical Guide to Applying Data Oriented Design: https://www.youtube.com/watch?v=IroPQ150F6c
Richard Fabian: Data-Oriented Design: https://www.dataorienteddesign.com/dodbook/
Array of Structs and Struct of Arrays
Well, then I just prompt Claude and get SIMD without having to learn it /s
I have no idea why you're being downvoted. HN has a fetish for SIMD, but if you are hand-rolling SIMD and you aren't writing an explicit acceleration library, you're doing it wrong. Like, 100% of the time.
Every modern language has a vectorization optimizing compiler, and through some fairly straightforward techniques this is automagic. And contrary to the various replies, unless you screwed something up compilers are really good at vectorizing on whatever hardware you're targeting, including SVE.
Every modern language has a vectorization optimizing compiler, and through some fairly straightforward techniques this is automagic. And contrary to the various replies, unless you screwed something up compilers are really good at vectorizing on whatever hardware you're targeting, including SVE.
> HN has a fetish for SIMD, but if you are hand-rolling SIMD and you aren't writing an explicit acceleration library, you're doing it wrong. Like, 100% of the time.
What if the “explicit acceleration library” for what you need to do doesn’t exist?
What if the “explicit acceleration library” for what you need to do doesn’t exist?
Or it exists and is not optimal for your use case?
Compilers are really good but really good is not actually that useful in cases where you need SIMD
I mean, utter bullshit. If you "need SIMD" you know exactly the programming pattern to guarantee SIMD from the compiler. And the single and only people who "need SIMD" know these rules. It is only the hobbyist "SIMD is neat" community that upvotes these ridiculous articles.
When I need SIMD I rarely use the compiler to help
Everyone should know about SIMD, so you can know when to ask your good friend Al, who knows how to do it, to use it for you in the right places.
I respect (and fear a little) those who intentionally utilize SIMD in their implementations, but I believe it's a bit too much of a semantic shift for 2-5x performance gain. Good news is that modern compilers are more than capable of emitting SIMD code even if original source is nothing but.
Most software's poor performance would be fixed long before SIMD comes into play. Reduce obvious DB/server round trips, bad data structure/cache locality, poor algorithm complexity, doing redundant computations, etc.
Most software's poor performance would be fixed long before SIMD comes into play. Reduce obvious DB/server round trips, bad data structure/cache locality, poor algorithm complexity, doing redundant computations, etc.
> Good news is that modern compilers are more than capable of emitting SIMD code even if original source is nothing but.
They're really not (I have a whole section on it in the blog post). This example in the post doesn't auto-vectorize, for example. And its a pretty big part of the overall throughput for plain text runs (ascii or unicode). Really, the point of that section is that almost nothing auto-vectorizes, backed up by LLVM docs and published research.
Instead, writing 12 lines for a 5x gain is way easier than crossing your fingers and hope someone else pays your bills.
Bigger picture, the real point is that this stuff isn't complicated. You wouldn't copy and paste 100 lines because you hope the compiler "lifts this into a for loop", you just write the for loop cause you know how and its simple.
Similarly, the common case of "process N values in parallel" is very simple. Write a dozen lines of code you're comfortable with. No need to pray the compiler people saved your bacon.
They're really not (I have a whole section on it in the blog post). This example in the post doesn't auto-vectorize, for example. And its a pretty big part of the overall throughput for plain text runs (ascii or unicode). Really, the point of that section is that almost nothing auto-vectorizes, backed up by LLVM docs and published research.
Instead, writing 12 lines for a 5x gain is way easier than crossing your fingers and hope someone else pays your bills.
Bigger picture, the real point is that this stuff isn't complicated. You wouldn't copy and paste 100 lines because you hope the compiler "lifts this into a for loop", you just write the for loop cause you know how and its simple.
Similarly, the common case of "process N values in parallel" is very simple. Write a dozen lines of code you're comfortable with. No need to pray the compiler people saved your bacon.
>Good news is that modern compilers are more than capable of emitting SIMD code even if original source is nothing but.
This is only true if you are intentionally writing code that the compiler can easily vectorise. Which is not most code.
This is only true if you are intentionally writing code that the compiler can easily vectorise. Which is not most code.
> Reduce obvious DB/server round trips
In my mental model of optimization, the above and SIMD are basically the same thing.
In my mental model of optimization, the above and SIMD are basically the same thing.
There is nothing to fear or respect about using SIMD. Don't lionize learning. You can learn too, if only you allow yourself to.
I just wouldn't start off with bold sentences as
> SIMD can be simple to understand
and
> writing SIMD is just about as easy as a for loop
and then the first example requires 12 lines to replace one line of scalar code.
Be honest and say SIMD is hard but the results are worth it!
(Another nitpick: if this article is for newbies, don't use SIMD-only words and concpts before explaining them. Step 5 is good: scalar tails are mentioned and described. Step 1 is bad: nobody is supposed to know what broadcast mean.)