Local async executors and why they should be the default(maciej.codes)
maciej.codes
Local async executors and why they should be the default
https://maciej.codes/2022-06-09-local-async.html
306 comments
I'm sympathetic to this point but I think that:
a) Saying Node + Deno are good is a stretch. Node has horrible performance, even for simple routing. And I'll source that[0].
b) Saying that adding `Send + Sync + 'static` bounds is a serious burden is, to me, overstating things.
> the far better model for writing performant servers.
It's completely workload dependent. For a chat server it's almost definitely not going to be more performant and you may end up with worse latency.
> it only costs you friction everywhere else in your entire codebase, and quite often performance as well.
I am unconvinced tbh. I do not believe that adding Send + Sync + 'static bounds is onerous, I do not believe satisfying those bounds is hard (it's almost always just a matter of moving the value), and I do not believe that the vast majority of programs benefit from TPC architecture.
I recognize that there is a problem here - that we are optimizing for one runtime at the expense of others - but I am not convinced at this point that the problem matters.
[0] https://www.techempower.com/benchmarks/#section=data-r21&tes...
a) Saying Node + Deno are good is a stretch. Node has horrible performance, even for simple routing. And I'll source that[0].
b) Saying that adding `Send + Sync + 'static` bounds is a serious burden is, to me, overstating things.
> the far better model for writing performant servers.
It's completely workload dependent. For a chat server it's almost definitely not going to be more performant and you may end up with worse latency.
> it only costs you friction everywhere else in your entire codebase, and quite often performance as well.
I am unconvinced tbh. I do not believe that adding Send + Sync + 'static bounds is onerous, I do not believe satisfying those bounds is hard (it's almost always just a matter of moving the value), and I do not believe that the vast majority of programs benefit from TPC architecture.
I recognize that there is a problem here - that we are optimizing for one runtime at the expense of others - but I am not convinced at this point that the problem matters.
[0] https://www.techempower.com/benchmarks/#section=data-r21&tes...
This is bad editorializing. You're putting words in the author's mouth that cannot even be found on that page.
Don't do that.
Edit: Thanks to the mods or whoever fixed it.
Don't do that.
Edit: Thanks to the mods or whoever fixed it.
> Yes the RwLock and mpsc comes from Tokio and lets you .await instead of blocking a thread, but these are not async primitives, these are multi-threading synchronization primitives.
The only reason all this async stuff even exists is because we want concurrency. We want to say "while this one task waits for I/O, this other task will do stuff". So it's not too surprising to me that an intro to async would include synchronization primitives. Those primitives aren't really "thread"-specific if by thread you mean OS thread. When you do async like this, you're basically re-implementing OS threads in user space.
The only reason all this async stuff even exists is because we want concurrency. We want to say "while this one task waits for I/O, this other task will do stuff". So it's not too surprising to me that an intro to async would include synchronization primitives. Those primitives aren't really "thread"-specific if by thread you mean OS thread. When you do async like this, you're basically re-implementing OS threads in user space.
"If you write regular synchronous Rust code, unless you have a really good reason, you don't just start with a thread-pool. You write single-threaded code until you find a place where threads can help you, and then you parallelize it, [..]"
I cannot agree more with that. As someone who's done a good deal of Java in my day job, I can tell you a thing or two about spawning threads willy-nilly. At least it is easier to avoid in Rust, but I'd still prefer it the other way round: opt-in, instead of opt-out.
I cannot agree more with that. As someone who's done a good deal of Java in my day job, I can tell you a thing or two about spawning threads willy-nilly. At least it is easier to avoid in Rust, but I'd still prefer it the other way round: opt-in, instead of opt-out.
> Making things thread safe for runtime-agnostic utilities like WebSocket is yet another price we pay for making everything multi-threaded by default. The standard way of doing what I'm doing in my code above would be to spawn one of the loops on a separate background task, which could land on a separate thread, meaning we must do all that synchronization to manage reading and writing to a socket from different threads for no good reason.
Why so? Libraries like quinn[1] define "no IO" crate to define runtime-agnostic protocol implementation. In this way we won't suffer by forcing ourselves using synchronization primitives.
Also, IMO it's relatively easy to use Send-bounded future in non-Send(i.o.w. single-threaded) runtime environment, but it's almost impossible to do opposite. Ecosystem users can freely use single threaded async runtime, but ecosystem providers should not. If you want every users to only use single threaded runtime, it's a major loss for the Rust ecosystem.
Typechecked Send/Sync bounds are one of the holy grails that Rust provides. Albeit it's overkill to use multithreaded async runtimes for most users, we should not abandon them because it opens an opportunity for high-end users who might seek Rust for their high-performance backends.
[1]: https://github.com/quinn-rs/quinn
Why so? Libraries like quinn[1] define "no IO" crate to define runtime-agnostic protocol implementation. In this way we won't suffer by forcing ourselves using synchronization primitives.
Also, IMO it's relatively easy to use Send-bounded future in non-Send(i.o.w. single-threaded) runtime environment, but it's almost impossible to do opposite. Ecosystem users can freely use single threaded async runtime, but ecosystem providers should not. If you want every users to only use single threaded runtime, it's a major loss for the Rust ecosystem.
Typechecked Send/Sync bounds are one of the holy grails that Rust provides. Albeit it's overkill to use multithreaded async runtimes for most users, we should not abandon them because it opens an opportunity for high-end users who might seek Rust for their high-performance backends.
[1]: https://github.com/quinn-rs/quinn
I find all the async stuff in Rust incredibly ugly, cumbersome, and its one of the biggest reasons I prefer C++, still. C++ lets me just write single- or multithreaded code, because none of the dependencies force their `async` stuff on me. Yeah, its up to me to ensure things are synchronized, but I'd rather do that than try to figure out how to get some dependency that isnt meant to use async to work in some async move closure.
The thing the article calls bad is not the thing the examples illustrate. The set-up is supposed to be that multithreading is a pain, but none of the examples actually do that. Take the initial list. "These are not async primitives, these are multi-threading synchronization primitives." Yeah, but they're multi-threaded forms of stuff you still need. If you needed RwLock in multi-threaded mode, you need RefCell in single-threaded mode, and the API is virtually identical. If your state needed to be in an Arc, it'll need to be in an Rc. And if you needed sync::mpsc, you'll instead need... sync::mpsc.
A Send bound is no great burden. The only type that you will regularly interact with that is not Send is the lock type from a Mutex or RwLock, which is good because if you hold it across a long await you can slow down your app, a bug prevention mechanism that does not exist in straight multithreading. The only point that actually illustrates a threading-caused problem is the thing about multithreading sockets, which it admits is almost imperceptible, and which you can also solve by not doing that.
Almost everything the author identifies as a parallelism problem is a 'static problem. Tokio is missing a scoped spawn like std has, and if it gained one then the much described multithreading woes would reduce to basically nothing.
A Send bound is no great burden. The only type that you will regularly interact with that is not Send is the lock type from a Mutex or RwLock, which is good because if you hold it across a long await you can slow down your app, a bug prevention mechanism that does not exist in straight multithreading. The only point that actually illustrates a threading-caused problem is the thing about multithreading sockets, which it admits is almost imperceptible, and which you can also solve by not doing that.
Almost everything the author identifies as a parallelism problem is a 'static problem. Tokio is missing a scoped spawn like std has, and if it gained one then the much described multithreading woes would reduce to basically nothing.
Can a mod fix the title please? The poster of this story has editorialized the title so bad that it has no connection with the actual title.
Actual title: Local Async Executors and Why They Should be the Default
Posted title: Async rust – are we doing it all wrong?
Really? Why this kind of terrible editorializing?
Actual title: Local Async Executors and Why They Should be the Default
Posted title: Async rust – are we doing it all wrong?
Really? Why this kind of terrible editorializing?
It's a frustrating area. As I've mentioned before, I'm writing a high-performance metaverse client in Rust, something which has many of the problems of both a web browser and a MMO. If you want to have a good looking metaverse, it takes a gamer PC multiple CPUs and a good GPU to deal with the content flood. (This is why Meta's Horizon looks so bad.) Now you have to use all that hardware effectively.
So what I'm writing uses threads. About 20 of them. They're doing different things at different priorities towards a coordinated goal. This is different from the two usual use cases - multiple servers running in the same address space, and parallel computation of array-type data.
Concurrency problems so far:
- Single-thread async is simple. Multi-thread async is complicated. Multi-thread async plus other threads not managed by the async system isn't used enough to be well supported.
- Rust is good at preventing race conditions, but it doesn't yet have a static deadlock analyzer. It needs one.
- Different threads at different priorities do work in both Linux and Windows, but not all that well. With enough low-priority compute-bound threads to keep all CPUs busy, high-priority threads do not get serviced in a timely manner. I was spoiled by previous work on QNX, which, being a true real-time operating system, takes thread priorities very seriously. On QNX, compute-bound background work has almost no effect on the high-priority stuff. Linux just doesn't work well at 100% CPU utilization. Unblocking a lock does not wake up a higher priority waiting thread immediately. This can delay high-priority threads unnecessarily.
- The WGPU crowd has spent a year getting their locking sorted out so that you can load content into GPU memory while the GPU is rendering something else. It's a standard feature of Vulkan graphics that you can do this, but it has to be supported at all levels above Vulkan too. For me, that's WGPU and Rend3. That stack is close enough to ready to test, but not ready for prime time yet.
- There's no way to cancel a pending HTTP request made with "ureq". "reqwest" supports that, but you have to bring in all the async and Tokio stuff, which means you now have multi-thread async plus other threads. This is only a problem for what I'm doing when the user closes the window, and the program needs to exit quickly and cleanly. I'm getting a 5-10 second stall at exit because of this.
- Crossbeam-channel is not "fair"; it's possible to starve out some requests. Parking-lot is fair, but doesn't have thread poisoning, which means that clean shutdowns after a panic are hard.
- Running under Wine with 100% CPU utilization with multiple threads results in futex congestion in Wine's memory allocation library, and performance drops by over 99%, with all CPUs stuck in spinlocks. The program is still running correctly, but at about 0.5 frames per second instead of 60 FPS. Bug reported and recognized by the Wine crew, but it's hard to fix. I can make this happen under gdb running my own code and see all those threads in the spinlocks, so I was able to file a good bug report. But I haven't generated a simple test case. It's a Wine-only problem; doesn't affect Microsoft Windows.
So that's life in a heavily threaded world.
Individual CPUs have not become much faster in over a decade. Everybody has been stuck at 3-4 GHz for a long time now. CPUs with many cores are widely available in everything from phones to game consoles. To use modern hardware effectively, you need threading.
So what I'm writing uses threads. About 20 of them. They're doing different things at different priorities towards a coordinated goal. This is different from the two usual use cases - multiple servers running in the same address space, and parallel computation of array-type data.
Concurrency problems so far:
- Single-thread async is simple. Multi-thread async is complicated. Multi-thread async plus other threads not managed by the async system isn't used enough to be well supported.
- Rust is good at preventing race conditions, but it doesn't yet have a static deadlock analyzer. It needs one.
- Different threads at different priorities do work in both Linux and Windows, but not all that well. With enough low-priority compute-bound threads to keep all CPUs busy, high-priority threads do not get serviced in a timely manner. I was spoiled by previous work on QNX, which, being a true real-time operating system, takes thread priorities very seriously. On QNX, compute-bound background work has almost no effect on the high-priority stuff. Linux just doesn't work well at 100% CPU utilization. Unblocking a lock does not wake up a higher priority waiting thread immediately. This can delay high-priority threads unnecessarily.
- The WGPU crowd has spent a year getting their locking sorted out so that you can load content into GPU memory while the GPU is rendering something else. It's a standard feature of Vulkan graphics that you can do this, but it has to be supported at all levels above Vulkan too. For me, that's WGPU and Rend3. That stack is close enough to ready to test, but not ready for prime time yet.
- There's no way to cancel a pending HTTP request made with "ureq". "reqwest" supports that, but you have to bring in all the async and Tokio stuff, which means you now have multi-thread async plus other threads. This is only a problem for what I'm doing when the user closes the window, and the program needs to exit quickly and cleanly. I'm getting a 5-10 second stall at exit because of this.
- Crossbeam-channel is not "fair"; it's possible to starve out some requests. Parking-lot is fair, but doesn't have thread poisoning, which means that clean shutdowns after a panic are hard.
- Running under Wine with 100% CPU utilization with multiple threads results in futex congestion in Wine's memory allocation library, and performance drops by over 99%, with all CPUs stuck in spinlocks. The program is still running correctly, but at about 0.5 frames per second instead of 60 FPS. Bug reported and recognized by the Wine crew, but it's hard to fix. I can make this happen under gdb running my own code and see all those threads in the spinlocks, so I was able to file a good bug report. But I haven't generated a simple test case. It's a Wine-only problem; doesn't affect Microsoft Windows.
So that's life in a heavily threaded world.
Individual CPUs have not become much faster in over a decade. Everybody has been stuck at 3-4 GHz for a long time now. CPUs with many cores are widely available in everything from phones to game consoles. To use modern hardware effectively, you need threading.
Multithreading, why does everyone always do it wrong?
One of life's big questions.
One of life's big questions.
Async is and probably will always be less usable than blocking Rust. It is a very, very useful mode of operating when you really need two of its biggest benefits: lightweight cooperative concurrency and task cancellation, but it comes at a big usability cost.
Rust software should use async tactically - in places where it is needed. Unfortunately handling http, which is a large part of many applications is actually a place where async has benefits. But if you plan to run your http behind nginx anyway (for TLS termination) even there using blocking http server might be a good idea.
> If you write regular synchronous Rust code, unless you have a really good reason, you don't just start with a thread-pool. You write single-threaded code until you find a place where threads can help you, and then you parallelize it,
I disagree with this one. When you work on a software project you should have the basic architecture figured out already, and main part of that is breaking your software into structurally parallel parts that can work independently. Adding ad-hoc parallelism after the fact works only for small scale things and will lead to rather accidental concurrency architecture.
Then for each part (groups of threads), figure out if it *needs* async. Between each part you'd communicate via channels or some shared data structures that rather easily can be made to work with both async/blocking code.
So e.g. an async http server, benefits from lightweight async concurrency, makes rpc-like channel-based calls to blocking IO / CPU/bussiness-logic intense workers (that don't benefit from async) where it makes sense. Each part is written in the best "type of Rust for its use-case".
Or if you need ability to cancel certain computations inside the larger framework (e.g. simulating agents etc.) you might want to nest async executor inside a blocking code.
Note: there's a lot of types of program archetypes out there (CRUD, ETL, data-intensive, embedded, frontent SPA, native mobile app) and I've noticed that many people are boxed in the type they happen to work on. CRUD applications (which are very common) are often 90% http handling-based and it might make sense to write them whole in async Rust.
Rust software should use async tactically - in places where it is needed. Unfortunately handling http, which is a large part of many applications is actually a place where async has benefits. But if you plan to run your http behind nginx anyway (for TLS termination) even there using blocking http server might be a good idea.
> If you write regular synchronous Rust code, unless you have a really good reason, you don't just start with a thread-pool. You write single-threaded code until you find a place where threads can help you, and then you parallelize it,
I disagree with this one. When you work on a software project you should have the basic architecture figured out already, and main part of that is breaking your software into structurally parallel parts that can work independently. Adding ad-hoc parallelism after the fact works only for small scale things and will lead to rather accidental concurrency architecture.
Then for each part (groups of threads), figure out if it *needs* async. Between each part you'd communicate via channels or some shared data structures that rather easily can be made to work with both async/blocking code.
So e.g. an async http server, benefits from lightweight async concurrency, makes rpc-like channel-based calls to blocking IO / CPU/bussiness-logic intense workers (that don't benefit from async) where it makes sense. Each part is written in the best "type of Rust for its use-case".
Or if you need ability to cancel certain computations inside the larger framework (e.g. simulating agents etc.) you might want to nest async executor inside a blocking code.
Note: there's a lot of types of program archetypes out there (CRUD, ETL, data-intensive, embedded, frontent SPA, native mobile app) and I've noticed that many people are boxed in the type they happen to work on. CRUD applications (which are very common) are often 90% http handling-based and it might make sense to write them whole in async Rust.
> If you know anything about asynchronous sockets it should be that multi-threading a socket doesn't actually yield you more requests / second, and it can actually lower it...
Re-read this a few times, and I'm fairly convinced it is not generally true. The author is also being a bit confusing about what exactly he means by "socket" here. Because while it's true that multi-threading over a server socket (e.g. the one that binds to the port when you launch the server) will not yield performance gains, multi-threading clients (that have their own sockets, including file descriptors) definitely will. That's the whole point of nginx thread pools[1]. Note that nginx does zero "CPU-bound" work, it literally just serves files.
Node/Deno being single-threaded is purely a limitation of Javascript. Tomcat, Jetty, etc. are all multi-threaded. I'm a bit tired, so I can't comment on the rest of the post in detail, but this was a bit of a red flag.
[1] https://gist.github.com/denji/8359866
Re-read this a few times, and I'm fairly convinced it is not generally true. The author is also being a bit confusing about what exactly he means by "socket" here. Because while it's true that multi-threading over a server socket (e.g. the one that binds to the port when you launch the server) will not yield performance gains, multi-threading clients (that have their own sockets, including file descriptors) definitely will. That's the whole point of nginx thread pools[1]. Note that nginx does zero "CPU-bound" work, it literally just serves files.
Node/Deno being single-threaded is purely a limitation of Javascript. Tomcat, Jetty, etc. are all multi-threaded. I'm a bit tired, so I can't comment on the rest of the post in detail, but this was a bit of a red flag.
[1] https://gist.github.com/denji/8359866
For me the biggest issue with Async is the management of multiple dependent async calls. It has some weird thing going on and I am not sure which pattern to use exactly. Some functions expect exactly same async fn signature some not and I am not sure why and which one to use.
Rust asyncs design was probably the mostly correct decision for what rust is: A general system programming language which you can use in most situations people today use C,C++ and more.
If rust would only be targeting web server programming the right decision might have been no async and green threads, it's just much easier to use.
But that rust would likely never have succeeded as most of it's initial success cases where for use-cases where you wouldn't want green threads.
Nicely we might still get no async and green threads: In form of run times which run WASM compiled rust code in a node like fashion. Probably in combination with some serverless/edge-compute providers which hopefully will be nice to use.
If rust would only be targeting web server programming the right decision might have been no async and green threads, it's just much easier to use.
But that rust would likely never have succeeded as most of it's initial success cases where for use-cases where you wouldn't want green threads.
Nicely we might still get no async and green threads: In form of run times which run WASM compiled rust code in a node like fashion. Probably in combination with some serverless/edge-compute providers which hopefully will be nice to use.
the title should be "Local Async Executors and Why They Should be the Default (Rust)"
While the article mostly focuses on the cognitive cost, which I deeply sympathize with, I do wonder about the runtime performance cost. Are there any good benchmarks actually quantifying the impact of all that extra thread-safety and the hoops that it adds? I'm not asking simply due personal interest in seeing the numbers, but also because if we want to steer the community towards this non-threadsafe direction it would help to have material to back the ideas and I suspect Rust community would be more responsive to complaints about perf than cognitive cost.
Moaning aside, what is (if any?) the direct equivalent to single-threaded asyncio, like Python or node.js, in Rust?
The thing I enjoy about async in Python is that it's very easy to write "thread"-safe code - you know exactly where you might give up execution context, and 90% of the time you have no need for mutexes and locks. But as this article complains, in Rust, it seems to be sync or multi-threaded.
The thing I enjoy about async in Python is that it's very easy to write "thread"-safe code - you know exactly where you might give up execution context, and 90% of the time you have no need for mutexes and locks. But as this article complains, in Rust, it seems to be sync or multi-threaded.
In C# i always wondered why they couldn't hide the async/await logic for most cases. I never need to fire off two IO futures at the same time, so just make the thread do other stuff if i'm waiting for IO feedback, don't make me type out async/await in all impacted functions, let the compiler figure out when it can process other stuff
> Posted on June 9, 2022
I'm still not sure what async (cooperative multitasking) gives over green threads (preemptive userspace multitasking)
When I was between jobs, I decided to learn rust by writing some async code. I really got stung and spent days doing things that would take minutes in C# or Node.
Part of the problem is that, in Rust, stack memory is easier to use than heap memory. When writing traditional threaded code, this isn't much of an issue because naturally most of our code is working with values on the stack.
BUT: When we look more closely in how async works in C# or Javascript, the compiler, under the hood, breaks up an async method into multiple methods and puts values that appear to be on the stack onto the heap. (Of course, I'm over simplifying.) It just works, and it just works well.
But, in Rust, making something async implicitly moves what appears to be on the stack onto the heap. It can quickly become hard to reason about.
I wish I knew about techniques that this article describes. Maybe it would make my code easier? In my current hobby projects, I'm doing traditional blocking IO because it's not "worth it" to write async. (In comparison, in Node, async code helps avoid nesting callbacks within callbacks, and in C#, doing IO in async is a best practice.)
Part of the problem is that, in Rust, stack memory is easier to use than heap memory. When writing traditional threaded code, this isn't much of an issue because naturally most of our code is working with values on the stack.
BUT: When we look more closely in how async works in C# or Javascript, the compiler, under the hood, breaks up an async method into multiple methods and puts values that appear to be on the stack onto the heap. (Of course, I'm over simplifying.) It just works, and it just works well.
But, in Rust, making something async implicitly moves what appears to be on the stack onto the heap. It can quickly become hard to reason about.
I wish I knew about techniques that this article describes. Maybe it would make my code easier? In my current hobby projects, I'm doing traditional blocking IO because it's not "worth it" to write async. (In comparison, in Node, async code helps avoid nesting callbacks within callbacks, and in C#, doing IO in async is a best practice.)
async/await is ugly and hard to use and understand for me. It is pretty reasonable that rust chose it because it is a zero-cost abstraction. But i just don't like it.
This article is against multithreaded executors by default and that synchronous code is easier to read and more practical than async code. I completely understand this.
My hobby and main interest is multithreading, async, coroutines, parallelism so I love articles like this so thank you.
I am trying to design a solution that lets us have our cake and eat it to. I want multithreaded coroutines or multithreaded async executors by default. I am trying to design a server and runtime that is largely parallel, concurrent, efficient, easy to understand, easy to reason about, async and easy to read and maintain. I want:
* Tokio is not bad but I think a codebase that uses async rust requires a high level of skill, cognitive load and understanding. It's not straightforward!
* synchronous straight-line flow code is the easiest to read and follow
* promises and callbacks aren't easy to read or follow control flow
* making a single threaded program parallel after writing it is almost a complete rewrite
* work stealing thread pools solve the starvation problem, so they're good!
* IO shouldn't block CPU and CPU shouldn't block IO
* I am trying to design a syntax that is data orientated that means programs can be parallelised after writing them, so parallelisation comes for free
* we can use the LMAX Disruptor pattern for efficient cross-thread communication. I use a lockfree multiconsumer multiproducer ringbuffer in my programs.
I have an epoll-server which multiplexes clients/sockets over threads, this is more efficient than a thread-per-socket/client. I need to change it into a websocket server.
Imagine you're a search engine company and you want to index links between URLs. How would you model this with async rust and thread pools?
`db.save()`, `download()` are IO intensive whereas `document.query("a")` and `parse` is CPU intensive.
I think its work diagram looks like this: https://github.com/samsquire/dream-programming-language/blob...
I've tried to design a multithreaded architecture that is scalable which combines lightweight threads + thread pools for work + control threads for IO epoll or liburing loops:
Here's the high level diagram:
https://github.com/samsquire/ideas5/blob/main/NonblockingRun...
The secret is modelling control flow as a data flow problem and having a simple but efficient scheduler.
I wrote about schedulers here and binpacking work into time:
https://github.com/samsquire/ideas4#196-binpacking-work-into...
I also have a 1:M:N lightweight thread scheduler/multiplexer:
https://github.com/samsquire/preemptible-thread
My hobby and main interest is multithreading, async, coroutines, parallelism so I love articles like this so thank you.
I am trying to design a solution that lets us have our cake and eat it to. I want multithreaded coroutines or multithreaded async executors by default. I am trying to design a server and runtime that is largely parallel, concurrent, efficient, easy to understand, easy to reason about, async and easy to read and maintain. I want:
* Tokio is not bad but I think a codebase that uses async rust requires a high level of skill, cognitive load and understanding. It's not straightforward!
* synchronous straight-line flow code is the easiest to read and follow
* promises and callbacks aren't easy to read or follow control flow
* making a single threaded program parallel after writing it is almost a complete rewrite
* work stealing thread pools solve the starvation problem, so they're good!
* IO shouldn't block CPU and CPU shouldn't block IO
* I am trying to design a syntax that is data orientated that means programs can be parallelised after writing them, so parallelisation comes for free
* we can use the LMAX Disruptor pattern for efficient cross-thread communication. I use a lockfree multiconsumer multiproducer ringbuffer in my programs.
I have an epoll-server which multiplexes clients/sockets over threads, this is more efficient than a thread-per-socket/client. I need to change it into a websocket server.
Imagine you're a search engine company and you want to index links between URLs. How would you model this with async rust and thread pools?
task download-url
for url in urls:
download(url)
task extract-links
parsed = parse(document)
return parsed
task fetch-links
for link in document.query("a")
return link
task save-data
db.save(url, link)
`db.save()`, `download()` are IO intensive whereas `document.query("a")` and `parse` is CPU intensive.
I think its work diagram looks like this: https://github.com/samsquire/dream-programming-language/blob...
I've tried to design a multithreaded architecture that is scalable which combines lightweight threads + thread pools for work + control threads for IO epoll or liburing loops:
Here's the high level diagram:
https://github.com/samsquire/ideas5/blob/main/NonblockingRun...
The secret is modelling control flow as a data flow problem and having a simple but efficient scheduler.
I wrote about schedulers here and binpacking work into time:
https://github.com/samsquire/ideas4#196-binpacking-work-into...
I also have a 1:M:N lightweight thread scheduler/multiplexer:
https://github.com/samsquire/preemptible-thread
Rust acolytes need to come to terms with the fact that if they want Rust to be the next C++, it's going to be the next C++. Heck, even C++ async is simpler. Anyway, who needs a package manager when you've got a package manager?
Suggesting that single threaded concurrency is the right way to do it when building tooling is completely asinine.
> why multi-threaded task executors should be the default
many many reasons and it's subtile and complicated
For one ironically it's just way easier to use, especially for less experienced programmers. In a task system like that it's just way easier to accidentally cause major issues when it's many tasks on one thread compared to multiple threads (at least with the guard rails rust provides for threading safety). On the other hand the performance overhead of by-default multi-threaded is just fine for a ton of use-cases to a point you could argue worrying about it is premature optimization.
Through it's important to state that a lot of this comes from the ecosystem around rust and not rust itself, as stuff like `LocalSet` shows you can have a non-multi threaded runtime and there is no reasons all the libraries you might use couldn't provide non multi thread safe versions. Some do. Just many decided that avoiding the performance overhead of being thread save isn't worth the maintenance overhead and additional foot guns it can provide.
Now naturally you can say "but node/deno/etc." but they are a completely different beast then "just" being not multi threaded by default. Like e.g. they don't have multi threaded code at all. Just single threaded code communicating through serialized messages (kinda). They also e.g. handle all I/O completely separated from you application code and don't have any non serialized communication between threads etc. etc.
Interestingly if you look at the design choices of the I/O Event loop (reactor) of tokio there are some conceptual similarities. Also there AFIK there is a company which is building something similar to the node model for rust using WASM.
I mean in the end the node-style approach is grate for building servers, but rust isn't just for building servers but much more general with much more low level use cases.
Now the main point where rust could improve quite a bit is to make it much easier to write a library which work very efficient with both cases. Code which crosses thread boundaries and code which doesn't. Currently you are often split between either implementing it twice, doing terrible unusable generics tricks or using tons of `cfg` (probably generated using macros/annotations) and hope no dependency accidentally imports the multi-thread feature when you don't need it. Non of this is really viable but it's a surprisingly hard problem. Currently the best idea I can come up for it is generic modules which make the "terrible to use generics tricks" usable, but it's probably not enough by a long stretch. (Even if a solution is found it might not work for Waker, even if it does you still might want sync/send Wakers in some cases.)
many many reasons and it's subtile and complicated
For one ironically it's just way easier to use, especially for less experienced programmers. In a task system like that it's just way easier to accidentally cause major issues when it's many tasks on one thread compared to multiple threads (at least with the guard rails rust provides for threading safety). On the other hand the performance overhead of by-default multi-threaded is just fine for a ton of use-cases to a point you could argue worrying about it is premature optimization.
Through it's important to state that a lot of this comes from the ecosystem around rust and not rust itself, as stuff like `LocalSet` shows you can have a non-multi threaded runtime and there is no reasons all the libraries you might use couldn't provide non multi thread safe versions. Some do. Just many decided that avoiding the performance overhead of being thread save isn't worth the maintenance overhead and additional foot guns it can provide.
Now naturally you can say "but node/deno/etc." but they are a completely different beast then "just" being not multi threaded by default. Like e.g. they don't have multi threaded code at all. Just single threaded code communicating through serialized messages (kinda). They also e.g. handle all I/O completely separated from you application code and don't have any non serialized communication between threads etc. etc.
Interestingly if you look at the design choices of the I/O Event loop (reactor) of tokio there are some conceptual similarities. Also there AFIK there is a company which is building something similar to the node model for rust using WASM.
I mean in the end the node-style approach is grate for building servers, but rust isn't just for building servers but much more general with much more low level use cases.
Now the main point where rust could improve quite a bit is to make it much easier to write a library which work very efficient with both cases. Code which crosses thread boundaries and code which doesn't. Currently you are often split between either implementing it twice, doing terrible unusable generics tricks or using tons of `cfg` (probably generated using macros/annotations) and hope no dependency accidentally imports the multi-thread feature when you don't need it. Non of this is really viable but it's a surprisingly hard problem. Currently the best idea I can come up for it is generic modules which make the "terrible to use generics tricks" usable, but it's probably not enough by a long stretch. (Even if a solution is found it might not work for Waker, even if it does you still might want sync/send Wakers in some cases.)
[deleted]
[deleted]
[deleted]
The whole thing just reads... ugly and inconsistent. It needs too much already-accumulated knowledge. As the article correctly points out, you need a bunch of stuff that are seemingly unrelated (and syntactically speaking you would never guess they belong together). And as other commenters pointed out, you need to scan a lot of docs -- many useful Tokio tools are not just not promoted, they are outright difficult to find at all.
Now don't get me wrong, I worked on projects where a single Rust k8s node was ingesting 150k events per second. I have seen and believed and I want to use Rust more. But the async story needs the team's undivided attention for a long time at this point, I feel.
Against my own philosophy and values I find myself attracted to Golang. It has a ton of ugly gotchas and many things are opaque... and I still find that more attractive than Rust. :(
This article is a sad reminder for me -- I am kind of drifting away from Rust. I might return and laugh at myself for this comment several months down the line... but at the moment it seems that my brain prefers stuff that's quicker to grok and experiment with. Not to mention writing demos and prototypes is objectively faster.
If I had executive power in the Rust leadership I'd definitely task them with taking a good hard look at the current state of async and start making backwards-incompatible changes (backed by new major semver versions of course). Much more macros or simply better-reading APIs might be a very good start. Start making and promoting higher-order concurrency and parallelism patterns i.e. the `scoped_pool` thingy for example.