The Zen of Parallel Programming(smolnero.com)
smolnero.com
The Zen of Parallel Programming
https://smolnero.com/posts/the-zen-of-parallel-programming
17 comments
There is no substance. It's navel-gazing dribble.
The "about" section of the website is equally vacuous:
The "about" section of the website is equally vacuous:
SmolNero is a philosophical startup rooted
in first-principles thinking.
We explore how humans relate to technology—
how we speak to it, depend on it, thank it,
and sometimes forget it’s there at all.
We don’t claim expertise. We see ourselves
as translators—working at the crossroads of
emotion and programming, curiosity and care
Seems to be run by a married couple of (maybe) burnt out account managers. That, in combination with some AI help, explains the empty "LinkedIn slop" writing style I think.
It's not often I comment on blog posts, but this is honestly one of the most meaningless blogs I've ever read (not counting those fully AI-generated SEO ones). I don't get why it ended up so high on Hacker News; it has an interesting title I guess, but pretty much no substance.
It's not often I comment on blog posts, but this is honestly one of the most meaningless blogs I've ever read (not counting those fully AI-generated SEO ones). I don't get why it ended up so high on Hacker News; it has an interesting title I guess, but pretty much no substance.
There have always been snake oil salesmen, for awhile they had to put the effort into knocking on your door peddling MLMs, now they're making more impressive snake oil, but vacuous, with LLMs.
> What sort of insight am I missing?
The article makes a lot more sense if you've practiced Zen.
Think of it as someone's way of making sense of Zen teachings by related them to something they already understand.
The article makes a lot more sense if you've practiced Zen.
Think of it as someone's way of making sense of Zen teachings by related them to something they already understand.
It is. The next post in the blog expands on the idea in concrete terms. The author doesn’t explicitly say so, but based on the shape of it I would assume all we have right now are the introduction and first full post of what is intended to be a multi-part series.
You are absolutely right.
It is just some daydreaming verbiage trying to connect totally disparate domains in a fanciful manner; nonsense and drivel.
As I continue reading An Introduction to Parallel Programming, I cannot help but notice a connection between communication among processors, communication among human beings, and communication within the individual self.
I say ... Whut?
It is just some daydreaming verbiage trying to connect totally disparate domains in a fanciful manner; nonsense and drivel.
As I continue reading An Introduction to Parallel Programming, I cannot help but notice a connection between communication among processors, communication among human beings, and communication within the individual self.
I say ... Whut?
Best quote over on distributed systems as applied to humans (from Dan Luu):
"Everything we've looked at so far is a technical problem. Compared to organizational problems, technical problems are straightforward. Distributed systems are considered hard because real systems might drop something like 0.1% of messages, corrupt an even smaller percentage of messages, and see latencies in the microsecond to millisecond range. When I talk to higher-ups and compare what they think they're saying to what my coworkers think they're saying, I find that the rate of lost messages is well over 50%, every message gets corrupted, and latency can be months or years"
- from https://danluu.com/sounds-easy/
"Everything we've looked at so far is a technical problem. Compared to organizational problems, technical problems are straightforward. Distributed systems are considered hard because real systems might drop something like 0.1% of messages, corrupt an even smaller percentage of messages, and see latencies in the microsecond to millisecond range. When I talk to higher-ups and compare what they think they're saying to what my coworkers think they're saying, I find that the rate of lost messages is well over 50%, every message gets corrupted, and latency can be months or years"
- from https://danluu.com/sounds-easy/
Fred Brooks and the mythical man month observed that communication between people expands faster than linear, and that adding more people to a project makes it later.
> Fred Brooks and the mythical man month observed that communication between people expands faster than linear, and that adding more people to a project makes it later.
In my experience with more than a couple dozen software projects, the ideal team size to successfully deliver a non-trivial effort is between five to nine people.
In my experience with more than a couple dozen software projects, the ideal team size to successfully deliver a non-trivial effort is between five to nine people.
One subject matter expert visionary
One technical architect visionary
One or two technical masters
One or two UI/UX masters
One or two padawans capable of becoming a master
I wonder how much a good process of meetings, tickets or whatever sort of documents etc for handling new ideas and features would speed this up or bring it back to linear. I've been in some orgs where good organization at the level above me can make the required communication between me and other dev teams lower and let us live in "good" silos
What really seems to make a difference is how well the project and the organization are at being split into indepedent groups.
I've been on projects where everybody needs to know everything, and on projects where many groups can be insulated from others.
The more your group can work without needing a meeting with other groups, the more things can progress in parallel.
Some projects won't work well with that constraint though. And some organizations won't either.
I've been on projects where everybody needs to know everything, and on projects where many groups can be insulated from others.
The more your group can work without needing a meeting with other groups, the more things can progress in parallel.
Some projects won't work well with that constraint though. And some organizations won't either.
I've heard that project architecture has a strong enough impact on feature completion rate that you can infer how tightly coupled vs loosely plugin based it is just from graphs of feature release.
It’s like distributed transactions, or cache-coherency protocols for CPUs.
Edit: aaaand I hadn’t read the article
Edit: aaaand I hadn’t read the article
I think it's kind of fun now. Agents can interact with Linear. I'm still playing around with like does an issue achieve anything? What is the fastest way to get a bug from the observing session to the originating session?
But it's secondary.
The wall is build. If you can AI program, your problem is the build is too slow, too unreliable, not secure enough from a supply chain standpoint.
That is where one competent senior hacker tops out today. The agents are yielding to a CI that gets 35% per-vCPU occupancy.
The wall right now is skill or build, depending on your skill.
But it's secondary.
The wall is build. If you can AI program, your problem is the build is too slow, too unreliable, not secure enough from a supply chain standpoint.
That is where one competent senior hacker tops out today. The agents are yielding to a CI that gets 35% per-vCPU occupancy.
The wall right now is skill or build, depending on your skill.
What are you talking about? A properly prompted agent generates a project with a fast, reproducible, and isolatable build. One of the best things about agentic AI is that I spend far less time on devops and on waiting on builds.
There are low-insensity regimes where it's all Python or whatever, and if you're in one, great.
When you're dealing with multiple platforms, or hardware accelerators, or mostly all of economically relevant shit in the AI era you don't get a small, clean, fast build.
Fable can't print a Tauri faux-native app without dragging in half of LLVM.
When you're dealing with multiple platforms, or hardware accelerators, or mostly all of economically relevant shit in the AI era you don't get a small, clean, fast build.
Fable can't print a Tauri faux-native app without dragging in half of LLVM.
For greenfield work, we are using mostly Rust, although small utilities built with Go or Python are also acceptable provided they have a comprehensive set of unit tests, an integration test suite, well-defined specifications and acceptance criteria, and thorough documentation. Once you have that, it's much easier for the AI to work on it. (Our main motivation for doing this is that it takes far fewer tokens now to get stuff done since we have all that, to the point I can leave Qwen 3.6 running in 24/7 loops accomplishing things.)
If your build is bloated and slow, it would seem prudent to go and fix that, possibly converting a codebase from some other language or build system to something that can be built and tested very quickly. Slow builds and slow unit tests are a choice.
If your build is bloated and slow, it would seem prudent to go and fix that, possibly converting a codebase from some other language or build system to something that can be built and tested very quickly. Slow builds and slow unit tests are a choice.
I'll contend that any Rust build involving Cargo is bloated and slow. `rustc` is impressively slow on a translation unit basis, and Cargo is basically a build recursion bingo card. Throw in about 900 micro point releases in flight at any given time?
If you're not running an elite `bazel` or `buck2` RBE with `nativelink`? You're not even playing.
If you're not running an elite `bazel` or `buck2` RBE with `nativelink`? You're not even playing.
I make sure projects are split into much smaller units so we don’t need some gargantuan bazel based build.
Having a bloated, barely-maintainable codebase and a build that takes hours and needs 100GB of storage is not a badge of honour.
Having a bloated, barely-maintainable codebase and a build that takes hours and needs 100GB of storage is not a badge of honour.
These days, adding more LLMs to a project is the interesting part http://replicated.live/blog/worktree
Amdahl's law says something similar, adding more processors to a problem eventually gives almost no speedup because parts of the problem cannot be parallelized and need to be done sequentially.
To the people wondering the meaning of the article, it's I think this.
To the people wondering the meaning of the article, it's I think this.
amdahl's law is trivial tautological nonsense and has nothing to do with the mythical man month.
The mythical man month is about communication between people and the amount of work to be productive in a project. It's about people colliding with programming.
It has nothing to do with a vague statement that "eventually" you have diminishing returns on processors (of course there is some limit even though different tasks could be many orders of magnitude apart).
The mythical man month is about communication between people and the amount of work to be productive in a project. It's about people colliding with programming.
It has nothing to do with a vague statement that "eventually" you have diminishing returns on processors (of course there is some limit even though different tasks could be many orders of magnitude apart).
Amdahl's law states diminishing returns are because some processes cannot be parallelized, thus eventually they become the bottleneck.
For Brooks the communication between people starts to dominate the time spent the more people you add, so it will eventually become the bottleneck.
For me it's an interesting parallel. Feel free to come up with your own conclusions.
For Brooks the communication between people starts to dominate the time spent the more people you add, so it will eventually become the bottleneck.
For me it's an interesting parallel. Feel free to come up with your own conclusions.
thus eventually they become the bottleneck
It's completely obvious and the "eventually" is where anything interesting happens. This comes from the early days of computers which is why something trivial became a supposed insight.
For Brooks the communication between people starts to dominate the time spent the more people you add, so it will eventually become the bottleneck.
Not just the communication between people while working in general but the communication to get someone involved in the project weighing down the people who would be making progress.
Feel free to come up with your own conclusions.
I did come up with own conclusion, the only similarity is the word bottleneck.
It's completely obvious and the "eventually" is where anything interesting happens. This comes from the early days of computers which is why something trivial became a supposed insight.
For Brooks the communication between people starts to dominate the time spent the more people you add, so it will eventually become the bottleneck.
Not just the communication between people while working in general but the communication to get someone involved in the project weighing down the people who would be making progress.
Feel free to come up with your own conclusions.
I did come up with own conclusion, the only similarity is the word bottleneck.
You try to speed up work by parallelizing, either by adding more machines or more humans. Seems pretty similar.
It might seem obvious if you don't think about it much, but one is a pragmatic breakdown from experience and one just says "adding cpus won't work forever if there is some part where you can't add more cpus".
One is basically 1 + 1 = 2 and the other is a warning of a non obvious situation that comes from lots of experience and effort.
One is basically 1 + 1 = 2 and the other is a warning of a non obvious situation that comes from lots of experience and effort.
It's still an interesting parallel, but that might be not be of interest if you're trying hard to be the middle guy from the bell curve meme.
It’s true that “zen of” is bolted onto many topics… it’s a tired cliche to suggest “simplicity” when things are complex. Does it help or is it faux spirituality?
I’ve enjoyed reading the comments here and I think there’s truth in how the technical problem is divided and teams are arranged. The idea of frequency of features (or builds) being a reflection of our division of the problem, is interesting. It made me think about our teams trying to ship releases and the problems arising, but zen and parallelism don’t give any hints. It’s just about effort to organise better, like it always was
I’ve enjoyed reading the comments here and I think there’s truth in how the technical problem is divided and teams are arranged. The idea of frequency of features (or builds) being a reflection of our division of the problem, is interesting. It made me think about our teams trying to ship releases and the problems arising, but zen and parallelism don’t give any hints. It’s just about effort to organise better, like it always was
(Speaking as a lapsed participant in a zen community. Opinions are my own, so others may disagree.)
The challenge for these “Zen and my job” types of work is that much of Zen practice is (deliberately) a bit of an inkblot test. If there’s any real Big Truth to it, it’s that it’s all one big muddle that everyone has to muddle through for themself. Unlike in many Western religious traditions, there is no presumption of univocality - even within the canonical literature, every opinion is understood to be the author’s alone and represents their own personal attempts at muddling things through. It’s just that some people and works are regarded as being particularly worth a read.
But that’s never explicitly stated. Which can be particularly confusing to people who grew up in religious traditions that do presume univocality and a greater degree of spiritual authority. Hence these works by people who see one particular bit that clicks for them and conclude, “Ah, this is what Zen is really about, I must go tell others!” without realizing that perhaps they just stumbled across an inkblot that they found to be particularly evocative.
The challenge for these “Zen and my job” types of work is that much of Zen practice is (deliberately) a bit of an inkblot test. If there’s any real Big Truth to it, it’s that it’s all one big muddle that everyone has to muddle through for themself. Unlike in many Western religious traditions, there is no presumption of univocality - even within the canonical literature, every opinion is understood to be the author’s alone and represents their own personal attempts at muddling things through. It’s just that some people and works are regarded as being particularly worth a read.
But that’s never explicitly stated. Which can be particularly confusing to people who grew up in religious traditions that do presume univocality and a greater degree of spiritual authority. Hence these works by people who see one particular bit that clicks for them and conclude, “Ah, this is what Zen is really about, I must go tell others!” without realizing that perhaps they just stumbled across an inkblot that they found to be particularly evocative.
I appreciate somebody trying to connect disparate concepts to give both some kind of fresh perspective; perfectly happy to entertain analogies that break down if you look at them too hard... but like somebody else said, this reads like a draft or a foreword. Also, what about any of this is "zen"? This goes nowhere and has no connection with the concepts put forward in the title. Chucking "the zen of" in front of anything for no reason was tired 10 years ago, yet people keep doing it. We get it, you've heard of the book.
This resonates with me. On the one hand I feel like my team is spread too thin all the time, scope and complexity is just too broad - and at the same time I'm pretty sure nore headcount could only improve it to a certain degree.
I like the part about synchronization and honesty. And I'm certain every time I get annoyed that someone is not as open with me as I wish they were, there is a part in there that myself contributed to that.
I like the part about synchronization and honesty. And I'm certain every time I get annoyed that someone is not as open with me as I wish they were, there is a part in there that myself contributed to that.
I try to find an embarrassingly parallel solution to most problems I encounter. Not only because such solutions scale, but because they often produce a simpler, more robust architecture which helps you to avoid future issues. It's great for avoiding single points of failure and performance chokepoints. Also, there is usually little to no overhead for choosing a parallelizable solution (besides a little bit of additional up-front thinking.)
To people who say "You don't need scalability" I say "You also don't need unscalability..."
To people who say "You don't need scalability" I say "You also don't need unscalability..."
CS101 Day 1: Learn about Divide and Conquer in depth. Honesty forms the ability for controlling signal strength, avoiding impedance mismatches or logically, synchronization between the divided slices.
Aside from the sentence "How many experiences continue to consume us because they were never allowed to finish burning?", I'm not sure what to make of this article. But the book it mentions, "Zen Mind, Beginner's Mind", seems worth taking a look at (an older one, not a promotion).
Yes, the article really only makes sense if you're both practiced Zen and done some intense parallel programming.
"Zen Mind, Beginner's Mind" makes more sense in a group setting. I read extracts of it with a Zen priest in a meditation group.
"Zen Mind, Beginner's Mind" makes more sense in a group setting. I read extracts of it with a Zen priest in a meditation group.
Communication in this sense is pretty common in Mathematics and Computer Science and it becomes apparent when its related to actors and their interactions.
A good example is game theory and game semantics. The term "actor" in this context are soo abstract that we can pretty much attempt to implement them anywhere we see fit and that I think is the beauty.
A good example is game theory and game semantics. The term "actor" in this context are soo abstract that we can pretty much attempt to implement them anywhere we see fit and that I think is the beauty.
Something that never gets old is Rob Pike's framing: concurrency is about structuring a program, while parallelism is about the execution. Plenty of concurrent programs never run in parallel and there are plenty of parallel programs that are poorly structured.
Zen of parallel programming AKA "Ditch the for loops"
These are your primitives:
https://gfxcourses.stanford.edu/cs149/fall25/lecture/datapar...
(e.g. CUDA or SQL)
These are your primitives:
https://gfxcourses.stanford.edu/cs149/fall25/lecture/datapar...
(e.g. CUDA or SQL)
Another parallel from the Western tradition: ‘homologia’, translated maybe best as congruence or coherence. A Greek Stoic concept of the mind/body/entire organism in alignment, caused by reason and feeling being in harmony.
There's a woman I read about a while back who did here PhD thesis on the notion that David Hume's contribution to the Enlightenment was inspired by conversations with a Jesuit monk who had returned to France from a long engagement in a Buddhist area of India. DDG is telling me it might be Allison Gopnik but I've no recollection of the name, only the contents.
She proved that he made a trip to a town where the monk lived after his return, and the library contained a copy of his reports. She could not prove more than opportunity but she strongly suspects that Hume not only read the report but may have met with the monk as well.
She proved that he made a trip to a town where the monk lived after his return, and the library contained a copy of his reports. She could not prove more than opportunity but she strongly suspects that Hume not only read the report but may have met with the monk as well.
I don't understand what you just wrote has to do with anything here.
I found elsewhere referring to her as an "independent philosopher", but maybe Rachel Paine? https://archive.philosophersmag.com/was-hume-a-secret-buddhi...
[deleted]
I'm genuinely considering that this is just a foreword and I missed the clickthrough link to the meat of the article.