Haiku package management(markround.com)
markround.com
Haiku package management
https://www.markround.com/blog/2023/02/13/haiku-package-management/
15 comments
Keep in mind that 'even before Fedora' was ~8 years ago
Not sure what my point is, I was just confused - I know SUSE/Fedora people are close
Not sure what my point is, I was just confused - I know SUSE/Fedora people are close
Do you know anything more about the solver? The GitHub page doesn’t really talk algorithms or anything. Does it use DPLL under the covers?
You can start here:
https://github.com/openSUSE/libsolv/blob/347200107f0f17a3697...
https://github.com/openSUSE/libsolv/blob/347200107f0f17a3697...
[deleted]
I'm genuinely so grateful that Haiku exists. The fact that a band of enthusiasts has managed to keep the spirit of BeOS alive after all these years is really inspiring. To me it's an echo of what felt like a more hopeful era in computing, when the internet still felt like a magical place.
I remember installing BeOS as a bare metal dual boot on my box circa 2001-2. My impression was: wow, this OS is fast, fun, and somehow endearing. I’ve not since had that experience with another OS.
I’m also super excited to see Haiku mature. Especially once it passes that breakaway threshold where it’s not primarily being run as a VM and the third-party ecosystem starts getting traction.
I’m also super excited to see Haiku mature. Especially once it passes that breakaway threshold where it’s not primarily being run as a VM and the third-party ecosystem starts getting traction.
Honestly that’s incredible. Is there anything close to it in the Linux world? Or I guess, I’m not an expert, that’s what NixOS is trying to solve?
>Just because they never got wide adoption doesn’t mean they were wrong
This is actually something that I was thinking about recently, applies to a lot of topics not just OSes
>Just because they never got wide adoption doesn’t mean they were wrong
This is actually something that I was thinking about recently, applies to a lot of topics not just OSes
Yeah, although Haiku seems to solve this problem more elegantly than Nix(the package manager) does(aka without a billion links). It is still great and I use it across many machines, including on OSX.
Another thing BeOS did well (and it's still ahead) was the filesystem. The mail program was implemented directly in the filesystem: https://arstechnica.com/information-technology/2018/07/the-b...
At one point in history, there was a chance that BeOS would be adopted as the Mac OS. I wonder how that fork in the road would look like.
Another thing BeOS did well (and it's still ahead) was the filesystem. The mail program was implemented directly in the filesystem: https://arstechnica.com/information-technology/2018/07/the-b...
At one point in history, there was a chance that BeOS would be adopted as the Mac OS. I wonder how that fork in the road would look like.
> Haiku seems to solve this problem more elegantly than Nix(the package manager) does(aka without a billion links). It is still great and I use it across many machines, including on OSX.
If Nix's solution were more elegant in this sense, you wouldn't be able to use it on macOS because macOS lacks the required sandboxing features.
If Nix's solution were more elegant in this sense, you wouldn't be able to use it on macOS because macOS lacks the required sandboxing features.
You’re right. That’s what NixOS is, a Linux distro where all software is managed by Nix, so the users environment is fully compartmentalized, declarative, and reproducible.
> so the users environment is fully compartmentalized, declarative, and reproducible.
I have a question about this, please forgive my ignorance. I see this statement repeated a lot about NixOS and I've kind of had some "success" in the comments section being "critical" / "asking hard questions" about NixOS.
If I have a build script that pulls + compiles + validates with hashes/checksums say, Linux kernel, some userspace tools, init system, etc. for a boring ol' distribution like Alpine or Debian, why isn't that classified as "reproducible" the same way NixOS is?
I have a question about this, please forgive my ignorance. I see this statement repeated a lot about NixOS and I've kind of had some "success" in the comments section being "critical" / "asking hard questions" about NixOS.
If I have a build script that pulls + compiles + validates with hashes/checksums say, Linux kernel, some userspace tools, init system, etc. for a boring ol' distribution like Alpine or Debian, why isn't that classified as "reproducible" the same way NixOS is?
The problem might not be your script, but that the system is stateful. Users can access and mutate the /usr directory, and some software will do this outside of the system's built-in package manager. You can also compile and link against a shared library in /usr, but then later this library may be updated, so now your app links against a different library than the one you built against.
Older package managers use a "name-version" scheme to try and avoid package conflicts, but in practice there can be different binaries which have the same "name-version" identifier and they're not always compatible.
The way Nix solves the issue is to store everything in a content-addressible store `/nix/store`, where multiple concurrent versions can be installed without naming conflicts (the directories are named with the content hash). When an app is compiled, it is linked to specific shared libraries in the nix store which will always be present on the system. The `/usr` directory then, is just a collection of symlinks to the actual content in the nix store (most recent versions), for compatibility.
Because derived packages are reproducible, they will always have the same content hash on every system on which they are deployed. As a developer then, you can target very specific shared libraries when building your software, and you know that when you deploy them, users are going to have the exact same dependencies you had when you built it.
Older package managers use a "name-version" scheme to try and avoid package conflicts, but in practice there can be different binaries which have the same "name-version" identifier and they're not always compatible.
The way Nix solves the issue is to store everything in a content-addressible store `/nix/store`, where multiple concurrent versions can be installed without naming conflicts (the directories are named with the content hash). When an app is compiled, it is linked to specific shared libraries in the nix store which will always be present on the system. The `/usr` directory then, is just a collection of symlinks to the actual content in the nix store (most recent versions), for compatibility.
Because derived packages are reproducible, they will always have the same content hash on every system on which they are deployed. As a developer then, you can target very specific shared libraries when building your software, and you know that when you deploy them, users are going to have the exact same dependencies you had when you built it.
> If I have a build script that pulls + compiles + validates with hashes/checksums say, Linux kernel, some userspace tools, init system, etc. for a boring ol' distribution like Alpine or Debian, why isn't that classified as "reproducible" the same way NixOS is?
It might be! Nix uses that very same strategy for enforcing reproducible outcomes when dealing with non-reproducible operations, like fetching things from the great mutating mass of the internet.
I would say that Nix differs from that approach mainly in the way it divides up the problem. So-called binary package managers like you use on Alpine and Debian perform runtime dependency resolution against a mutating global namespace that usually lives on the internet— the binary repos themselves. Generally, those repos include only one copy of each package. So to halt the forward march of all those versions, you have to mirror each repo of binary artifacts you're interested in using.
With Nix, outcomes also depend on the state of the repo, but the repo is just a collection of build recipes which are the mapped onto their outputs in a single, huge binary cache. Every new build gets added to the same binary cache, for every Nixpkgs/NixOS release. It just grows and grows. And of course, Nix builds work even without enabling the binary cache at all.
So to get the same kind of reproducibility out of many other package managers, you generally have to set up additional infrastructure.
But if you do everything right, you can certainly have a fully reproducible OS image based on Alpine or whatever else.
It might be! Nix uses that very same strategy for enforcing reproducible outcomes when dealing with non-reproducible operations, like fetching things from the great mutating mass of the internet.
I would say that Nix differs from that approach mainly in the way it divides up the problem. So-called binary package managers like you use on Alpine and Debian perform runtime dependency resolution against a mutating global namespace that usually lives on the internet— the binary repos themselves. Generally, those repos include only one copy of each package. So to halt the forward march of all those versions, you have to mirror each repo of binary artifacts you're interested in using.
With Nix, outcomes also depend on the state of the repo, but the repo is just a collection of build recipes which are the mapped onto their outputs in a single, huge binary cache. Every new build gets added to the same binary cache, for every Nixpkgs/NixOS release. It just grows and grows. And of course, Nix builds work even without enabling the binary cache at all.
So to get the same kind of reproducibility out of many other package managers, you generally have to set up additional infrastructure.
But if you do everything right, you can certainly have a fully reproducible OS image based on Alpine or whatever else.
Nix's main innovation with regard to reproducibility is that it actually fully specifies the dependency graph. I'm not going to say anything about bit reproducibility as that's another thing entirely, but virtually every other package manager and build system has holes in its dependency graph, coming in 2 main forms:
1. Implicit dependencies that are needed to build the software but not managed by the package manager or build system itself. For example, if your language package manager depends on a C library that it can't build itself. Or Linux distribution package managers that implicitly assume the presence of a "base system".
2. SAT solvers for version selection. Using one of these usually introduces an implicit dependency on the clock time, because your dependencies will resolve to different versions at different times.
The reason your script wouldn't classify as reproducible is simply that you would probably fail to fully specify your dependencies. It's harder than you'd expect once you start moving up the software stack to complicated high-level applications.
1. Implicit dependencies that are needed to build the software but not managed by the package manager or build system itself. For example, if your language package manager depends on a C library that it can't build itself. Or Linux distribution package managers that implicitly assume the presence of a "base system".
2. SAT solvers for version selection. Using one of these usually introduces an implicit dependency on the clock time, because your dependencies will resolve to different versions at different times.
The reason your script wouldn't classify as reproducible is simply that you would probably fail to fully specify your dependencies. It's harder than you'd expect once you start moving up the software stack to complicated high-level applications.
It's only reproducible if you can actually reproduce it in a different environment (different time/locale/machine/etc).
It would be, the problem is that you can not write such a script at the moment. As far as I know those checksums are almost always for the source code/file itself and not the compiled binary.
The situation has improved, actually - you'd still have to do custom scripting to check the output checksums AFAIK but ex. https://tests.reproducible-builds.org/debian/reproducible.ht... tracks Debian's attempts to make it possible to reproduce packages perfectly. It's just that nixos made it pervasive and normal through the whole system.
I think the main thing to understand is that Debian is reproducible as a consequence of significant unenforced contracts around what the pristine build environment will be, what the state of the package repo will be when dependencies are fetched and installed, and so on (see: https://www.debian.org/doc/debian-policy/ch-source.html#repr...). Your local dpkg-buildpackage invocation has pretty much a 0% chance of being a bit-for-bit reproduction of what you pulled from the archive unless you cooperate with all this ceremony.
Nix, on the other hand, is reproducible by default. If you `nix build <nixpkgs/gitsha#pkgname>` you are automatically getting a local build that takes place in the same hermetic sandbox as the build you pulled from cache.nixos.org, and will be identical or very close to it. (Contrary to popular belief, not all of nixpkgs is reproducible— many packages require patching for various sources of nondeterminism, and this is an ongoing effort: https://github.com/NixOS/nixpkgs/labels/6.topic%3A%20reprodu...)
Nix, on the other hand, is reproducible by default. If you `nix build <nixpkgs/gitsha#pkgname>` you are automatically getting a local build that takes place in the same hermetic sandbox as the build you pulled from cache.nixos.org, and will be identical or very close to it. (Contrary to popular belief, not all of nixpkgs is reproducible— many packages require patching for various sources of nondeterminism, and this is an ongoing effort: https://github.com/NixOS/nixpkgs/labels/6.topic%3A%20reprodu...)
many compilers aren't guaranteed to be fully deterministic tho (e.g. __DATE__ macro)
The "Reproducable Builds" project has been working on this for a few years now, and introduced the SOURCE_DATE_EPOCH environment variable, which is supported by most compilers, as a way to fix the date in build environments.
Other previously nondeterministic build issues have some similar opt-in workarounds. Have a look around the Reproducable Builds project website for more info/examples.
https://reproducible-builds.org/docs/source-date-epoch/
https://gcc.gnu.org/onlinedocs/cpp/Environment-Variables.htm...
Other previously nondeterministic build issues have some similar opt-in workarounds. Have a look around the Reproducable Builds project website for more info/examples.
https://reproducible-builds.org/docs/source-date-epoch/
https://gcc.gnu.org/onlinedocs/cpp/Environment-Variables.htm...
> Honestly that’s incredible. Is there anything close to it in the Linux world?
Yeah, this is similar in spirit to what Flatpak and Snap do. The latter even uses SquashFS images it mounts for every package.
I have a feeling, though, that this is more similar to Distri¹, where the package manager is a bit more traditional. That is to say, packages are fine-grained (not packaging large runtimes into single chunks, like Flatpak and Snap) and there's not as much special tooling as Flatpak has for redirecting inputs and outputs, sandboxing, etc.
Distri also eliminates all hooks and triggers from packages, leading to super fast installs, which is pretty cool.
> Or I guess, I’m not an expert, that’s what NixOS is trying to solve?
From the article, three key features highlighted are:
But spiritually I'd say the basic idea (avoid dependency hell through lightweight package isolation) is similar and so are the results, at least in terms of capabilities.
PS: If you're interested in another example, RedoxOS' package manager also has a very similar design to Haiku's and Distri's, IIRC.
—
1: https://distr1.org/
2: Because NixOS allows you to switch to a new system without booting it (i.e., it lets you upgrade without rebooting), it actually has two symlinks here: booted-system and current-system. The current-system symlink rotates as soon as you switch configurations, but booted-system gets set only on a new boot.
Yeah, this is similar in spirit to what Flatpak and Snap do. The latter even uses SquashFS images it mounts for every package.
I have a feeling, though, that this is more similar to Distri¹, where the package manager is a bit more traditional. That is to say, packages are fine-grained (not packaging large runtimes into single chunks, like Flatpak and Snap) and there's not as much special tooling as Flatpak has for redirecting inputs and outputs, sandboxing, etc.
Distri also eliminates all hooks and triggers from packages, leading to super fast installs, which is pretty cool.
> Or I guess, I’m not an expert, that’s what NixOS is trying to solve?
From the article, three key features highlighted are:
> - Immutable system directories
> - Rollback to previous states
> - User-managed packages separated from system packages
But spiritually I'd say the basic idea (avoid dependency hell through lightweight package isolation) is similar and so are the results, at least in terms of capabilities.
PS: If you're interested in another example, RedoxOS' package manager also has a very similar design to Haiku's and Distri's, IIRC.
—
1: https://distr1.org/
2: Because NixOS allows you to switch to a new system without booting it (i.e., it lets you upgrade without rebooting), it actually has two symlinks here: booted-system and current-system. The current-system symlink rotates as soon as you switch configurations, but booted-system gets set only on a new boot.
It's kind of what snap is trying to be, but it's not there yet.
See my comment about apkovl
Been watching Haiku for around a decade now, install it every year or so to give it a go and it is getting very close to being my primary OS. Great project with a good community, devs are very helpful.
Same for me. It inches closer every release.
Another great feature of Haiku!
I love playing from time to time with it. If there would be ARM64 support (I wish I could find time to contribute...), combined with the progress of Asahi Linux, it would be fun (and a pun to history) to run Haiku on Apple M1s!
I love playing from time to time with it. If there would be ARM64 support (I wish I could find time to contribute...), combined with the progress of Asahi Linux, it would be fun (and a pun to history) to run Haiku on Apple M1s!
Author of the article here. Thanks for posting this! There's also a really interesting thread on lobste.rs [1] where a Haiku developer talks more about some of their less user-visible innovations, compares the Linux development model to Haiku, and gives some background on the drama[2] caused by the adoption of their package management system.
Well worth a read and fascinating (to me, anyway!) to see new ideas and approaches from the Alt-OS scene.
[1]=https://lobste.rs/s/j9decl/haiku_package_management
[2]=https://lobste.rs/s/j9decl/haiku_package_management#c_owqi3t
Well worth a read and fascinating (to me, anyway!) to see new ideas and approaches from the Alt-OS scene.
[1]=https://lobste.rs/s/j9decl/haiku_package_management
[2]=https://lobste.rs/s/j9decl/haiku_package_management#c_owqi3t
That was good article :D
Definitely going to look into Haiku more myself :)
Definitely going to look into Haiku more myself :)
I highly recommend it! Even if you haven't had the experience or attachment to BeOS it's a refreshing and fun take on a desktop system. It just "clicks" with me in a way that Linux/Windows/Mac never did. It's truly impressive that they've been going for over 20 years now (I still remember the first screenshot of app_server drawing a window in the early 2000s) and have accomplished so much from scratch. Can't wait to see where the project will go next!
Haiku and Serenity
Two founts of hope
For computing's future
Two founts of hope
For computing's future
Serenity is just a Unix clone and not one written in an interesting language either. Id put redox above it simply for being Rust based. Haiku is an entirely different paradigm.
Serenity's starting point is very well described:
"SerenityOS is a love letter to '90s user interfaces with a custom Unix-like core. It flatters with sincerity by stealing beautiful ideas from various other systems.
Roughly speaking, the goal is a marriage between the aesthetic of late-1990s productivity software and the power-user accessibility of late-2000s *nix."
I think in terms of paradigm, Serenity has a stronger ideology than RedoxOS. It brings BeOS and Mac concepts like a consistent and fully integrated monorepo system and development kits, and a very strong and defined user interface paradigm.
From that perspective I find Redox much less of a paradigm for the end-user, if the main feature is the language the developers used. Orbital does not look remarkable.
The level of productivity of Andreas himself, his videos and the community he managed to built, including growing a few individuals of to major contributors to the point they created a whole browser stack is just outstanding as an open-source case study. Also to me, much more interesting than a programming language used to built it.
Haiku is written in the same language as Serenity, btw.
"SerenityOS is a love letter to '90s user interfaces with a custom Unix-like core. It flatters with sincerity by stealing beautiful ideas from various other systems.
Roughly speaking, the goal is a marriage between the aesthetic of late-1990s productivity software and the power-user accessibility of late-2000s *nix."
I think in terms of paradigm, Serenity has a stronger ideology than RedoxOS. It brings BeOS and Mac concepts like a consistent and fully integrated monorepo system and development kits, and a very strong and defined user interface paradigm.
From that perspective I find Redox much less of a paradigm for the end-user, if the main feature is the language the developers used. Orbital does not look remarkable.
The level of productivity of Andreas himself, his videos and the community he managed to built, including growing a few individuals of to major contributors to the point they created a whole browser stack is just outstanding as an open-source case study. Also to me, much more interesting than a programming language used to built it.
Haiku is written in the same language as Serenity, btw.
What I tried to convey in my Haiku but better described!
I'd add to that, Andreas is creating his own memory-safe compiled language to build components for the OS. Jakt has already been used to build things, including its own compiler.
Also I think what makes both HaikuOS and Serenity compelling is the common vision and clear direction: Aiming to recreate then improve BeOS has given HaikuOS project a clear steer ahead, while Andreas' clear guidelines and monorepo has done the same for SerenityOS.
I'd add to that, Andreas is creating his own memory-safe compiled language to build components for the OS. Jakt has already been used to build things, including its own compiler.
Also I think what makes both HaikuOS and Serenity compelling is the common vision and clear direction: Aiming to recreate then improve BeOS has given HaikuOS project a clear steer ahead, while Andreas' clear guidelines and monorepo has done the same for SerenityOS.
As a teenager reading "PC Pro" magazine BeOS was the operating system I was convinced was going to be the next thing and take over. Sad times!
Looks a lot like apkovl https://wiki.alpinelinux.org/wiki/Alpine_local_backup for Alpine linux. Not the most easy to grasp documentation but it's basically a package (apk, similar do rpm or deb) that each has its own overlayfs.
I swear, pet project operating systems like Haiku and SerenityOS are the only things that still fill me with hope and keep me going these days. I could go on for hours writing love letters to Haiku.
the hacker me
Haiku was my first experience writing code for a real world software organisation, as part of Google Summer of Code.
Glad to see it still thriving and making the news every once in a while.
Glad to see it still thriving and making the news every once in a while.
My Linux research distribution “distri” works the same way :)
(I learnt about Haiku’s similar approach after building distri.)
https://michael.stapelberg.ch/posts/2019-08-17-introducing-d...
https://distr1.org/
(I learnt about Haiku’s similar approach after building distri.)
https://michael.stapelberg.ch/posts/2019-08-17-introducing-d...
https://distr1.org/
I'm a big fan of distri! Really enjoyed your blog posts on its design and development. Hopefully I didn't misremember too much about it in these comments. :)
Spent some time digging through the website today figuring out where Haiku stands. Seems they still have not sorted out the issues with non-native file systems, it is getting better but still buggy and some (most?) are read only. This is the number one issue that is keeping me from adopting Haiku as my primary system, It looks like I can now access all of my external drives with Haiku but digging through the site has not left me confident enough to let Haiku have at my drives.
Other than that it seems solid as long as your hardware is well supported. Think I will commit my laptop to it this weekend and see how long it lasts.
Other than that it seems solid as long as your hardware is well supported. Think I will commit my laptop to it this weekend and see how long it lasts.
(Haiku developer here.)
I would trust the NTFS driver pretty implicitly at this point both for read/write; it's based on NTFS-3G, and following the rewrite I did of the Haiku-specific parts of it in late 2021, appears to be very solid in testing.
The EXT2/3/4 driver is also very good, but there may be some issues remaining there. However I haven't heard reports of it destroying data.
The FAT driver, on the other hand, has some cases where it has been known to corrupt directories. It could probably use a rewrite; most of its code dates to the 1990s...
Those are the "big 3" of other filesystems, anyway. I don't think there's a status page for all of them.
I would trust the NTFS driver pretty implicitly at this point both for read/write; it's based on NTFS-3G, and following the rewrite I did of the Haiku-specific parts of it in late 2021, appears to be very solid in testing.
The EXT2/3/4 driver is also very good, but there may be some issues remaining there. However I haven't heard reports of it destroying data.
The FAT driver, on the other hand, has some cases where it has been known to corrupt directories. It could probably use a rewrite; most of its code dates to the 1990s...
Those are the "big 3" of other filesystems, anyway. I don't think there's a status page for all of them.
Thanks for the heads up. The only current issue with ext4 that I found was https://dev.haiku-os.org/ticket/16392 which makes me a little nervous and I don't think I will be plugging in my primary backup drive just yet but at least it can now deal with journaling on ext4, progress. For now I can deal without that but this is the thing that has historically caused me to take Haiku off the computer.
ZFS and Btrfs both are very good these days?
Can linux mount the Haiku version of BFS? Seem to recall that this was an issue the first few times I tried Haiku and it was buggy and unreliable but don't quite remember or recall trying it since. Searches are not pulling up anything useful. If not is this something Haiku is planing to work on or are you just going to let linux take care of that when ever they get around too it?
ZFS and Btrfs both are very good these days?
Can linux mount the Haiku version of BFS? Seem to recall that this was an issue the first few times I tried Haiku and it was buggy and unreliable but don't quite remember or recall trying it since. Searches are not pulling up anything useful. If not is this something Haiku is planing to work on or are you just going to let linux take care of that when ever they get around too it?
> The only current issue with ext4 that I found was https://dev.haiku-os.org/ticket/16392
A 3 year old ticket with only one reproduction from 2 years ago, and it appears to have been fixed in the interim, and now the ticket has been closed by the main developer of the ext2/3/4 driver. I wouldn't be too worried about that.
> I don't think I will be plugging in my primary backup drive just yet
That's probably a good idea anyway.
> but at least it can now deal with journaling on ext4, progress.
I think it's supported journaling for many years, I don't believe that to be a new feature.
> ZFS and Btrfs both are very good these days?
We don't support ZFS at all, and btrfs support is read-only (there's experimental write support but it's disabled in default builds still, I think.)
> Can linux mount the Haiku version of BFS?
Yes, read-only with the upstream Linux driver. There is a way to get read-write support with Haiku's driver under FUSE, however it's not very performant to say the least.
A 3 year old ticket with only one reproduction from 2 years ago, and it appears to have been fixed in the interim, and now the ticket has been closed by the main developer of the ext2/3/4 driver. I wouldn't be too worried about that.
> I don't think I will be plugging in my primary backup drive just yet
That's probably a good idea anyway.
> but at least it can now deal with journaling on ext4, progress.
I think it's supported journaling for many years, I don't believe that to be a new feature.
> ZFS and Btrfs both are very good these days?
We don't support ZFS at all, and btrfs support is read-only (there's experimental write support but it's disabled in default builds still, I think.)
> Can linux mount the Haiku version of BFS?
Yes, read-only with the upstream Linux driver. There is a way to get read-write support with Haiku's driver under FUSE, however it's not very performant to say the least.
FWIW, it looks like they just closed that ticket now. I guess another developer saw this comment.
Interestingly, some niche Linux distributions, such as Puppy Linux and Porteus mount packages which are SquashFS filesystems at runtime, implementing exactly this paradigm long before Nix and the likes were around.
This reminds me of TinyCore Linux, which distributes packages as squashfs archives and dynamically mounts them at runtime with symlinks to the system directories. Pretty neat.
[deleted]
Is the package system there something that Haiku invented, or was BeOS like this as well? I had a BeOS install at one point in the 90s, but don't remember how that worked.
It's a Haiku thing. Added around 2018 if I recall correctly - see the linked lobste.rs thread in another of my comments where a Haiku developer gives some (really interesting) background on the development and effects on the community.
libsolv, created by Michael Schoeder at SUSE, has two innovations:
- a memory optimized format to represent packages and its dependencies, using hashed string pools - a SAT solver able to operate directly in this representation, battle tested in this particular scenario of complex upgrades, including hundred of testcases.
Haiku uses mainly the solver only, as far as I saw in the source.