Thanks! Actually, the mentioned PR was just an internal merge; the one to the master branch is this one https://github.com/apache/iggy/pull/2299 which happened last week :)
Sans-IO is for clients only, and we'd like to introduce it one day to SDK to support the different runtimes as well as the blocking, sync client.
There's a core difference between the message queue and the stream, and one of them is that the append-only log acts as a simple database from which multiple independent consumers can read records at the same time; hence, you don't truncate the append-only log unless specified in its settings via a particular cleanup policy (e.g. based on message expiry or its allowed size).
Right, I meant TCP without TLS, and when it comes to QUIC I only did benchmark using the "dangerous" mode (disabled local cert validation). I don't have the exact numbers, but AFAIR on MacOS raw TCP was a few times faster than QUIC, while on Linux (e.g. PopOS distro) TCP was maybe around 20-30% faster?
Still, there are plenty of options to be configured in Quinn (buffer size, send/receive window etc.), so it might be that with some tweaking the performance would be similar or QUIC would be even faster than TCP - this is also one of the things that I'd like to spend more time with in the future. We expose most of these options in the server configuration file, so it's easy to adjust.
Thank you for contributing to such a great library, really easy to work with :)
I did some dotnet OSS in the past, but there's something special about Rust community and I guess that if I'd picked another language instead, I wouldn't gather such an amazing team :)
As already mentioned, we've decived to use monoio, just to have something to begin with, it will take at least a few months to rewrite the core parts to make use of io_uring and thread-per-core approach (if feasible), so staying on the bleeding edge is more like a sandbox for us - we might decide to go with another runtime, like glommio, mfio or something else.
Of course, they hide the details behind io_uring, it's just that monoio seemed even easier to work with. It's not set in stone, though, actually there are new runtimes being developed as we speak, for example mfio - we'll see what will be our final choice, but we've decided to start with monoio.
Speaking of the other platforms, having the fallback to epoll or kqueue (they've even announced some Windows integration) is nice to have, however, at some point we might just purely focus on Linux development anyway (io_uring only), if there'd be any issues e.g. when it comes to the code design to provide the compatibility across multiple OS.
Can't say for sure, we share some of the same concepts that can be found in Kafka or RabbitMQ Streams, but at the same time, try to build something from the ground up and focus on low-level I/O improvements :)
I had 0 Rust experience and close to 0 of system programming experience (except playing with some lower level communication protocols etc. but nothing fancy), but I'd say that something like message streaming isn't the system programming - it's close to building the database, but nowhere close to making direct usage of low level system APIs.
I've started with QUIC, as I wanted to try out something new. Still, TCP protocol we have in place is a bit faster than QUIC, but it might be due to the lack of some additional tuning or so. On a side note, QUIC on MacOS is slow, when compared to Linux.
This is a message streaming infrastructure, similar to Kafka, Redpadna etc. You can think of it as a kind of a simple database (so-called Write Ahead Log), being responsible for appending the messages in a highly performant manner.