> though I hope existing well tested rust libraries are used rather than NIHing them and introducing new bugs
I think that's much more likely to introduce bugs.
Think of it that way, a lot of the Rust libraries are rewriting existing copyleft libraries in permissive licenses, so they cannot look at the original code, dooming them to repeat the mistakes that were made in the original code and having to fix them all over again on their own (as both go from "oh this is simple" to "oh another corner case").
I just want to translate code 1:1 to Rust, reusing my existing knowledge, design decisions, and tests. It should behave _exactly_ the same as before, just memory safe.
Like the last one was/is the inability to comprehend safety of large buffers on ppc64el because the stack clash protector code generated by gcc isn't understood. The one before that was more problems of that sort on armhf where it also didn't understand the clash protector - in more cases.
It's quite surprising and it takes days to weeks to debug each of these, going down to the assembler level and verifying that by hand.
As far as I understand, Fil-C changes the ABI of the system, therefore it requires a new architecture in Debian terminology, e.g. amd64fil. And then you'd need to use multi-arch to pull in amd64fil binaries where that works.
We'll have to see how this plays out but it's not super plug and play.
It's certainly what we aim for in APT. We do have an overwrite of course, since we need to copy uninitiated data around: The cache file is allocated as a whole and written at the end, but not all parts of it are used, but it triggers stuff.
Don't want to introduce complex code to only copy the parts that are actually reachable would be silly and introduce bugs.
But keep in mind valgrind is super buggy and we spend quite a bunch of time working around valgrind false positives (outside of amd64)
It's not like I'm in a hurry to switch to Rust and will spend full steam on it. It's amongst the lowest priority items.
A lot of the Rust rewrites suffer a crucial issue: they want a different license than what they are rewriting and hence rewrite from scratch because they can't look at the code.
But here we're saying: Hey we have this crucial code, there may be bugs hidden in it (segfaults in it are a recurring source of joy), and we'll copy that code over from .cc to .rs and whack it as little as possible so it compiles there.
The problem is much more there on the configuration parser for example which does in a sense desparately need a clean rewrite, as it's way too sloppy, and it's making it hard to integrate.
In an optimal world I'd add annotations to my C++ code and have a tool that does the transliteration to Rust at the end; like when the Go compiler got translated from C to Go. It was glorious.
It's insanely complex, particularly you want _verified_ crypto. Last year (or two years ago?) I had to fix a tiny typo in OpenSSL's ARM assembly for example, it was breaking APT and Postgres left and right, but only got triggered on AWS :D
> > since it only supports amd64 at this time and is maintained by a single genius.
> That's easily fixable.
as easily as fixing Rust to work on the remaining 4 architectures?
> > It also doesn't help you to attract new contributors.
> I don't understand this point.
C++ doesn't attract a lot of developers, Rust attracts many more. I want more community, particularly _young_ community. I don't wanna work on this alone all the time :D
Do keep in mind that a lot of the people involved in these sorts of things are neurodiverse in some ways, and may have significant trouble dealing with change.
As teh64 helpfully pointed out in https://news.ycombinator.com/item?id=45784445 some hours ago, 4ish years ago my position on this was a total 360 and I'd have had the same reaction to now-me's proposal.
Isn't it also funny that all of these things are done by the same person?
In all seriousness though, let me assure you that I plan to take a very considerate approach to Rust in APT. A significant benefit of doing Rust in APT rather than rewriting APT from scratch in Rust means that we can avoid redoing all our past mistakes because we can look at our own code and translate it directly.
Keep in mind APT is using the GNU dialect of C++, particularly C++ 17, shortly C++ 23. And it always exploits the latest C++ features while at the same time still having workarounds in the code for a lack of namespaces in C++ compilers prior to the C++ standardization...
Actual cryptography code, the best path is formally verified implementations of the crypto algorithms; with parsers for wrapper formats like OpenPGP or PKCS#7 implemented in a memory safe language.
You don't want the core cryptography implemented in Rust for Rust's sake when there's a formally verified Assembler version next to it. Formally verified _always_ beats anything else.
If anyone sees that horrible mess of hacks around pre-STL C++'s lacks of namespace in combination with latest C++ features as part of the C++ community I'd be very surprised :D
If APT were a hardcore C++ project surely we'd have like adopted namespaces everywhere by now.
I wish, but I get new security bugs in those components like every year or so, not all are tracked with security updates to be fair, some we say it's your own fault if you use the library to parse untrusted code.
After all the library wasn't designed around safety, we assumed the .debs you pass to it are trusted in some way - because you publish them to your repository or you are about to install them so they have root maintainer scripts anyway.
But as stuff like hosting sites and PPAs came up, we have operators publishing debs for untrusted users, and hence suddenly there was a security boundary of sorts and these bugs became problematic.
Of course memory safety here is only one concern, if you have say one process publishing repos for multiple users, panics can also cause a denial of service, but it's a step forward from potential code execution exploits.
I anticipate the rewrites to be 1 to 1 as close as possible to avoid introducing bugs, but then adding actual unit tests to them.
Fil-C is amazing but is much more problematic than Rust at this point since it only supports amd64 at this time and is maintained by a single genius.
It also doesn't help you to attract new contributors. With the changes we made over in Ubuntu to switch to rust-coreutils and sudo-rs, we have seen an incredible uptake in community contributions amongst other things, and it's very interesting to me to try to push APT more into the community space.
At this time, most of the work on APT is spent by me staying awake late, or during weekends and my 2 week Christmas break, the second largest chunk is the work I do during working hours but that's less cool and exciting stuff :D
Adding Rust into APT is one aspect; the other, possibly even more pressing need is rewriting all the APT documentation.
Currently the APT manual pages are split into apt-get and apt-cache and so on, with a summary in apt(8) - we should split them across apt install(8), apt upgrade (8) and so on. At the same time, DocBook XML is not very attractive to contributors and switching to reStructuredText with Sphinx hopefully attracts more people to contribute to it.
It's rather that GnuPG is ill-regarded for its code immaturity tbh. You don't even need to read the code base, just try to use it in a script:
It exits 0 when the verification failed, it exits 1 when it passed, and you have to ignore it all and parse the output of the status fd to find the truth.
It provides options to enforce various algorithmic constraints but they only work in some modes and are silently ignored in others.
I think that's much more likely to introduce bugs.
Think of it that way, a lot of the Rust libraries are rewriting existing copyleft libraries in permissive licenses, so they cannot look at the original code, dooming them to repeat the mistakes that were made in the original code and having to fix them all over again on their own (as both go from "oh this is simple" to "oh another corner case").
I just want to translate code 1:1 to Rust, reusing my existing knowledge, design decisions, and tests. It should behave _exactly_ the same as before, just memory safe.