Re: Integrating "safe" languages into OpenBSD? (2017)(marc.info)
marc.info
Re: Integrating "safe" languages into OpenBSD? (2017)
https://marc.info/?l=openbsd-misc&m=151233345723889&w=2
389 comments
It also doesn't help that Rust keeps on pushing on the idea that "static linking is the only way to go". This is another cargo-cult which I wish didn't end up being engrained so deep in the toolchain because while it has some merits, it also has significant drawbacks of a typical unix distribution.
Static linking might be good for folks distributing a single server binary over a fleet of machines (pretty much like Go's primary use case), or a company that only cares about a single massive binary containing the OS itself (the browser), but it stops "being cool" very quickly on a typical *nix system if you plan to have hundreds of rust tools lying around.
Size _matters_ once you have a few thousands binaries on your system (mine has short of 6k), think about the memory requirements of caching vs cold-starting each, and running those and how patching for bug or vulnerabilities will pan out. There's a reason dynamic linkage was introduced and it's _still_ worth it today.
And rebuilding isn't cheap in storage requirements either: a typical cargo package will require a few hundred megabytes of storage, likely for _each_ rebuild.
Two days ago I rebuilt the seemingly innocuous weechat-discord: the build tree takes about 850mb on disk, with a resulting binary of 30mb. By comparison, the latest emacs binary from git rebuilds in a 1/10 of the time with all the features enabled, the build tree weights 150mb (most of which are lisp sources+elc) with a results in a binary of 9mb, not stripped.
Static linking might be good for folks distributing a single server binary over a fleet of machines (pretty much like Go's primary use case), or a company that only cares about a single massive binary containing the OS itself (the browser), but it stops "being cool" very quickly on a typical *nix system if you plan to have hundreds of rust tools lying around.
Size _matters_ once you have a few thousands binaries on your system (mine has short of 6k), think about the memory requirements of caching vs cold-starting each, and running those and how patching for bug or vulnerabilities will pan out. There's a reason dynamic linkage was introduced and it's _still_ worth it today.
And rebuilding isn't cheap in storage requirements either: a typical cargo package will require a few hundred megabytes of storage, likely for _each_ rebuild.
Two days ago I rebuilt the seemingly innocuous weechat-discord: the build tree takes about 850mb on disk, with a resulting binary of 30mb. By comparison, the latest emacs binary from git rebuilds in a 1/10 of the time with all the features enabled, the build tree weights 150mb (most of which are lisp sources+elc) with a results in a binary of 9mb, not stripped.
Let's try to summarize what he tries to say but kinda doesn't do very well:
_Rust and OpenBSD are not a good fit._
OpenBSD keeps to certain aspects about how to do thinks which just don't work well with rust. This doesn't mean they are better or worse. They are just different.
For example outside of rustc-dev hardly anyone compiles rust them self. As such not many work on making rust compile itself on exotic systems (and yes in 2020 i386 is a exotic system, even through it was the standard in the past). I mean improving cross compilation is for now much more important (as far as I can tell).
But OpenBSD requires the ability to self compile and support hardware for a very long time so rust (for now) doesn't fit.
Another example is that many rewrites of classical small tools provide little value to anyone _if_ they are 100% compatible. As such they tend _to not be compatible_ in favor of better interfaces/output etc. (E.g. ripgrep!!)
Lastly OpenBSD is stuck in a world where thinks move very slow. But this is also why many people use it. One the other hand rust is from a world where thinks move much faster, sure with small steps and backward compatibility but a small step every 6 weeks is still a lot over a fiew years. So they just don't fit well together.
I also believe that in the future they might fit well together at some point. But not for now and probably not in the next 3 or so years.
_Rust and OpenBSD are not a good fit._
OpenBSD keeps to certain aspects about how to do thinks which just don't work well with rust. This doesn't mean they are better or worse. They are just different.
For example outside of rustc-dev hardly anyone compiles rust them self. As such not many work on making rust compile itself on exotic systems (and yes in 2020 i386 is a exotic system, even through it was the standard in the past). I mean improving cross compilation is for now much more important (as far as I can tell).
But OpenBSD requires the ability to self compile and support hardware for a very long time so rust (for now) doesn't fit.
Another example is that many rewrites of classical small tools provide little value to anyone _if_ they are 100% compatible. As such they tend _to not be compatible_ in favor of better interfaces/output etc. (E.g. ripgrep!!)
Lastly OpenBSD is stuck in a world where thinks move very slow. But this is also why many people use it. One the other hand rust is from a world where thinks move much faster, sure with small steps and backward compatibility but a small step every 6 weeks is still a lot over a fiew years. So they just don't fit well together.
I also believe that in the future they might fit well together at some point. But not for now and probably not in the next 3 or so years.
> I wasn't implying. I was stating a fact. There has been no attempt to move the smallest parts of the ecosystem, to provide replacements for base POSIX utilities.
This is in fact incorrect--there is a project aiming to build all of the coreutils in Rust (https://github.com/uutils/coreutils).
More to the point: while I do concur in the conclusion that Rust shouldn't be a part of the OpenBSD base system, the gatekeeping implied here (it's not a serious language because it's not used to build an operating system) is really toxic. Especially considering that the gate in question has already been thoroughly breached by the language in question (some universities have switched to using Rust in their OS courses, for example).
This is in fact incorrect--there is a project aiming to build all of the coreutils in Rust (https://github.com/uutils/coreutils).
More to the point: while I do concur in the conclusion that Rust shouldn't be a part of the OpenBSD base system, the gatekeeping implied here (it's not a serious language because it's not used to build an operating system) is really toxic. Especially considering that the gate in question has already been thoroughly breached by the language in question (some universities have switched to using Rust in their OS courses, for example).
As with most things Theo says, they're aligned with being the right answer for OpenBSD's niche, which is probably not your niche.
Any new tools that have high resource needs, don't support all the more exotic/esoteric platforms that it supports, are all nonstarters if your goal is a very small core system that compiles quickly on all supported platforms.
Any new tools that have high resource needs, don't support all the more exotic/esoteric platforms that it supports, are all nonstarters if your goal is a very small core system that compiles quickly on all supported platforms.
A rewrite of core utils[1] was started in 2013.
[1]: https://github.com/uutils/coreutils.git
[2]: https://www.redox-os.org/
commit d4e96b33e343733992fad55ac840c9649cd72ede
Author: Jordi Boggiano <[email protected]>
Date: Fri Aug 2 09:22:57 2013 -0700
Initial commit
[1]: https://github.com/uutils/coreutils.git
[2]: https://www.redox-os.org/
> However there is a rampant fiction that if you supply a new safer
method everyone will use it. For gods sake, the simplest of concepts
like the stack protector took nearly 10 years for adoption, let people should switch languages? DELUSION.
Yeah, I agree: if the sole value proposition were 'security' it would be a really slow roll. cargo is the killer app IMO. Anyone who's walked a big dependency tree - download, build, oh oops this requires libfoo - download, build, oh oops that requires libbar...
But I think lots of coders land on rust because they want or need something faster or more portable than Python or JS and rust is 'easier' than C.
Yeah, I agree: if the sole value proposition were 'security' it would be a really slow roll. cargo is the killer app IMO. Anyone who's walked a big dependency tree - download, build, oh oops this requires libfoo - download, build, oh oops that requires libbar...
But I think lots of coders land on rust because they want or need something faster or more portable than Python or JS and rust is 'easier' than C.
I have been using bat and exa along with alacritty for a while. All of them are pretty good
https://github.com/alacritty/alacritty
https://github.com/sharkdp/bat
https://github.com/ogham/exa
This is from 2017.
In 2020 I personally use ag 99% of the time, and when I do use grep is because its options are burned in my brain and it saves me 30 seconds by not looking up the equivalent for ag.
https://github.com/ggreer/the_silver_searcher
In 2020 I personally use ag 99% of the time, and when I do use grep is because its options are burned in my brain and it saves me 30 seconds by not looking up the equivalent for ag.
https://github.com/ggreer/the_silver_searcher
Marc.info seems not to be responding for me. Here is an archive.org link. https://web.archive.org/web/20190302233853/https://marc.info...
> Adding such a toolchain would take make build time
from 40 minutes to hours. I don't see how that would happen.
This should have been at the top of the email because this is the main issue (along with the i386 one). As other people have noted in this thread some people have already re-written basic utilities in Rust (and other languages).
Unless someone manage to make the Rust compiler much much faster then it is currently it won't get accepted into OpenBSD anytime soon.
This should have been at the top of the email because this is the main issue (along with the i386 one). As other people have noted in this thread some people have already re-written basic utilities in Rust (and other languages).
Unless someone manage to make the Rust compiler much much faster then it is currently it won't get accepted into OpenBSD anytime soon.
Such ecosystems come with incredible costs. For instance, rust cannot even compile itself on i386 at present time because it exhausts the address space.
Is this true anymore or just a history note?
Is this true anymore or just a history note?
This is not "de Raadt on Rust". There isn’t any de Raadt’s opinion on Rust.
This is Theo de Raadt on integrating Rust utilities in OpenBSD.
This is Theo de Raadt on integrating Rust utilities in OpenBSD.
Interesting read and raises one big issue:
We need to make memory safe languages accessible and with that, the tools and other frameworks for a new language isn't an easy path.
Which gets to an aspect in many walks of life - if you are doing something from scratch - you can use the latest knowledge and enforce that, but knowledge moves on and bringing up the rest of the system inline with a what is needed to use that new language/knowledge is not an overnight task, it's a slow process - and even if you provide something better, you can not guarantee it will be embraced at a needed pace to keep the momentum going.
Much like anything in life, you have better trains than you did when they was invented, but legacy infrastructure precludes an instant/overnight switch, due to impact upon what is used currently.
So case of, requirements for something better like rust support, is a bit of a catch-22 as you need it there to get the people to use it so tools get written, but to get it there you kinda need the tools so that people use it to write the tools. Hence the first step is always the biggest.
Which makes you wonder how intertwined aspects of the OS and programs are, even with standards like POSIX, there is clearly a bigger larger standard needed.
This along with the dependances and you can see why docker and other application level virtualization has much going for it.
We need to make memory safe languages accessible and with that, the tools and other frameworks for a new language isn't an easy path.
Which gets to an aspect in many walks of life - if you are doing something from scratch - you can use the latest knowledge and enforce that, but knowledge moves on and bringing up the rest of the system inline with a what is needed to use that new language/knowledge is not an overnight task, it's a slow process - and even if you provide something better, you can not guarantee it will be embraced at a needed pace to keep the momentum going.
Much like anything in life, you have better trains than you did when they was invented, but legacy infrastructure precludes an instant/overnight switch, due to impact upon what is used currently.
So case of, requirements for something better like rust support, is a bit of a catch-22 as you need it there to get the people to use it so tools get written, but to get it there you kinda need the tools so that people use it to write the tools. Hence the first step is always the biggest.
Which makes you wonder how intertwined aspects of the OS and programs are, even with standards like POSIX, there is clearly a bigger larger standard needed.
This along with the dependances and you can see why docker and other application level virtualization has much going for it.
> For instance, rust cannot even compile itself on i386 at present time because it exhausts the address space.
Wow, that one was really cruel.
Wow, that one was really cruel.
(2017)
Perhaps relevant: Ted Unangst recently wrote some Rust code:
https://flak.tedunangst.com/post/reliverator
https://flak.tedunangst.com/post/reliverator
Working on an operating system that uses some of these modern / safer alternatives, the concerns around bootstrapping, lack of spec, lack of formal memory model, long compile times, and larger output binaries are valid. They are real and present challenges for us, that we choose to work against. It is currently extremely difficult to produce a binary of the size of bsd cat using these technologies, and very large amounts of work are needed to close that gap, if it even can be.
In a new system you may not need cat though, but if you're a unix system with a heavy shell and textual basis, you do need lightweight versions of these things - light to build, light to run, and well integrated into the environment.
What's the path out for unix systems? I have no idea, but at least as of today I have not seen a modern safe language that is ready to replace unix use cases. Maybe zig? It's hello world is at least under 10kb.
In a new system you may not need cat though, but if you're a unix system with a heavy shell and textual basis, you do need lightweight versions of these things - light to build, light to run, and well integrated into the environment.
What's the path out for unix systems? I have no idea, but at least as of today I have not seen a modern safe language that is ready to replace unix use cases. Maybe zig? It's hello world is at least under 10kb.
I want to like the rust language. The syntax is nice and the features are interesting.
My problem is the ecosystem. The packaging and build system clearly isn’t designed to accommodate classic desktop applications. The FFI is really painful, and even Firefox has tons of wrapper code to just integrate with the libraries it needs to have a GUI.
In that respect I agree with Theo in that Rust developers are focusing on far too narrow of a use case, mostly web facing applications or data processing. Cargo being so similar to npm kind of confirms this. Rust isn’t intended for me.
My problem is the ecosystem. The packaging and build system clearly isn’t designed to accommodate classic desktop applications. The FFI is really painful, and even Firefox has tons of wrapper code to just integrate with the libraries it needs to have a GUI.
In that respect I agree with Theo in that Rust developers are focusing on far too narrow of a use case, mostly web facing applications or data processing. Cargo being so similar to npm kind of confirms this. Rust isn’t intended for me.
Feels like I've just had a coke. Nice reply from Theo.
He might just be wrong: https://github.com/uutils/coreutils
The only reason he might still be right is if they don't aim to be POSIX-compliant. The page doesn't explicitly call out compliance as a goal, but given that it has some busybox tests, it's probably compliant or very close.
The only reason he might still be right is if they don't aim to be POSIX-compliant. The page doesn't explicitly call out compliance as a goal, but given that it has some busybox tests, it's probably compliant or very close.
Rust is good at preventing memory leaks. It doesn't matter (as much) for a command line utility that runs for one second then exits. Of course, there are now utilities written in Rust. https://github.com/samuela/rustybox
There's something to it, but I don't thing the future is as grim as that.
Rust is a stepping stone. It is a proof of concept that shows a lot of problems can be solved at compile time without adding huge bloat at runtime.
Yes, Rust has problems. Some of them are just because the language is new and you feel not yet fully built.
The compile times are more more insidious side effect of Rust's properties. I think, realistically, it will take some time but the problems will be resolved either in future Rust or in another language based on it. And it will be very worth the wait.
High level languages are fun and important, but it is not possible that all system code is going to be built in high level languages. As we run stuff that is more and more demanding of CPU we need a good low level language that will also tick some important checks of reliability.
Rust is a stepping stone. It is a proof of concept that shows a lot of problems can be solved at compile time without adding huge bloat at runtime.
Yes, Rust has problems. Some of them are just because the language is new and you feel not yet fully built.
The compile times are more more insidious side effect of Rust's properties. I think, realistically, it will take some time but the problems will be resolved either in future Rust or in another language based on it. And it will be very worth the wait.
High level languages are fun and important, but it is not possible that all system code is going to be built in high level languages. As we run stuff that is more and more demanding of CPU we need a good low level language that will also tick some important checks of reliability.
Interesting points. Would a rustc + cranelift solution manage to boostrap itself in a 32-bit address space?
Theo is being conservative here. It may just save OpenBSD some 10 years from now when Rust is no longer in the wind due to Hooch. At that point, someone will propose rewriting coreutils in Hooch, yet it may be much more sensible to stick with C, or rather C29.
BeOS/Haiku got itself into the 250 MB range of a full-featured distro, in part by emphasizing code reuse in C++. (Though my understanding is that they used GNU grep et. al.) The current macroeconomic climate is wrong for this, but what if someone funded an effort to base an implementation of an entire OS on Rust, let's say, using Linux as a bootstrap? So there would be no requirements for "base to build base," and the developers dogfooding the toolchain would be strongly motivated to make builds faster, which would be rolled back into Rust.
Please put a (2017) in title.
This was discussed before here:
https://news.ycombinator.com/item?id=15843631
I think it would have been good to include (2017) in the title. 2-3 years is quite a long time in the area of technology, and in my view, Rust has come a long way since 2017
(2017)
There isn't really much of a statement or judgment on Rust. At most there's an interesting point on it's value proposition:
> However there is a rampant fiction that if you supply a new safer method everyone will use it. For gods sake, the simplest of concepts like the stack protector took nearly 10 years for adoption, let people should switch languages? DELUSION.
This is mostly true. Developers in particular don't generally care about security, so selling Rust as a "secure" language is not going to be enough. I've said this since 1.0. But it's not entirely true for products, which often drive development - Chrome's pairing of performance and security led to tons of marketing wins.
Given that tools like "cat" etc are:
a) Not generally security sensitive
b) Target developers
I don't see anyone choosing the "rust cat" over the builtin. This is why people build tools like "bat", that aren't just memory safe copies, but they're memory safe tools that add features and target slightly different cases.
Not much else to get from this post, I think.