Super Colliding Nix Stores: Nix Flakes for Millions of Developers(blog.replit.com)
blog.replit.com
Super Colliding Nix Stores: Nix Flakes for Millions of Developers
https://blog.replit.com/super-colliding-nix-stores
78 comments
Finding a specific version of a package is also really difficult. It requires you to find the commit hash in the Nixpkgs repo that last contained it, and there isn't a central search engine that lets you map version to commit.
We ended up building package version search into the Devbox CLI for this reason (https://www.jetpack.io/devbox/docs/guides/pinning_packages/#...). There's also a 3rd party site that lets you search for Nix package versions (https://lazamar.co.uk/nix-versions/)
We ended up building package version search into the Devbox CLI for this reason (https://www.jetpack.io/devbox/docs/guides/pinning_packages/#...). There's also a 3rd party site that lets you search for Nix package versions (https://lazamar.co.uk/nix-versions/)
Normally I would eye-roll at these kind of self promotions but it looks like you are trying to help with a real pain point of Nix.
I put a good amount of time getting to grips with "raw" nix with the I imagine common yo-yo-ing between "I don't get it" and "oh I see, this is great" but when I realised how the intersection of nixpkgs and package versioning actually worked.. I was done.
For a tool that from the outside seems is so heavily focused on immutability to just continually throw away old versions in nixpkgs head is a head scratcher, and as a monorepo isn't a great fit for utilising different revisions for different packages either.
I can only guess that due to single repo containing every package it wasn't seen as practical to just continually append versions to, but when the diff log is just full of 'delete version X URL and it's hash, add X+1 and new hash' from the outside at least, it felt like a real missed opportunity.
I put a good amount of time getting to grips with "raw" nix with the I imagine common yo-yo-ing between "I don't get it" and "oh I see, this is great" but when I realised how the intersection of nixpkgs and package versioning actually worked.. I was done.
For a tool that from the outside seems is so heavily focused on immutability to just continually throw away old versions in nixpkgs head is a head scratcher, and as a monorepo isn't a great fit for utilising different revisions for different packages either.
I can only guess that due to single repo containing every package it wasn't seen as practical to just continually append versions to, but when the diff log is just full of 'delete version X URL and it's hash, add X+1 and new hash' from the outside at least, it felt like a real missed opportunity.
I would suggest one more step of "oh, this is great": I don't think people need to care as much about finding the right version in the repo as they do. (Unless we're talking about something really significant like finding old MySQL 5.6) For smaller apps you can either override or copy the current nix file and update the version and hash - you get the version you want. It will use new version of the dependencies, but normally that's just fine.
I've got 3 packages which are pinned to a specific version that way in home-manager and I'm happy. It's not an approach for the first time user of course.
I've got 3 packages which are pinned to a specific version that way in home-manager and I'm happy. It's not an approach for the first time user of course.
I thought this too, but we got a lot of requests for better version search + pinning. Adding overrides and looking up the right commit hash is a pretty cumbersome for a first time user.
For new projects though, I agree that using `latest` is generally the way to go.
For new projects though, I agree that using `latest` is generally the way to go.
I believe the Hackage.nix library [1] has a solution to this problem, specifically for Haskell packages.
The `text` package [2] has many major, minor, and fix versions. Many of these versions are still required by projects. However, the current Nix Packages only has one version [3], and maybe a second version under a different name.
Hackage.nix will provide any version of `text` published on Hackage.
One downside is that `text` is not pulled from Nix Packages. The project's specific version of `text` must be cached elsewhere, or built from source. So it is slower.
[1] https://github.com/input-output-hk/hackage.nix [2] https://hackage.haskell.org/package/text [3] https://search.nixos.org/packages?channel=22.11&show=haskell...
The `text` package [2] has many major, minor, and fix versions. Many of these versions are still required by projects. However, the current Nix Packages only has one version [3], and maybe a second version under a different name.
Hackage.nix will provide any version of `text` published on Hackage.
One downside is that `text` is not pulled from Nix Packages. The project's specific version of `text` must be cached elsewhere, or built from source. So it is slower.
[1] https://github.com/input-output-hk/hackage.nix [2] https://hackage.haskell.org/package/text [3] https://search.nixos.org/packages?channel=22.11&show=haskell...
> just continually throw away old versions in nixpkgs head
It is simply completely infeasible to store and build every version of every package.
While it is not user friendly to find a package’s version X, it is either just overriding the version field and the hash, or referencing another version of the nixpkgs tree and building the same package from there.
It’s not trivial with other package managers either, and on top you can easily get into inconsistent states with those — nix can handle any version of any software correctly.
It is simply completely infeasible to store and build every version of every package.
While it is not user friendly to find a package’s version X, it is either just overriding the version field and the hash, or referencing another version of the nixpkgs tree and building the same package from there.
It’s not trivial with other package managers either, and on top you can easily get into inconsistent states with those — nix can handle any version of any software correctly.
I don't think the answer is to have every version at the nixpkg head, for exactly the reasons you mentioned.
But I do think there's a search issue where mapping the version field of a package to the commit hash or nixpkgs version is not trivial or user-friendly, especially for multiple packages.
But I do think there's a search issue where mapping the version field of a package to the commit hash or nixpkgs version is not trivial or user-friendly, especially for multiple packages.
I do think Flakes help with the "throwing away old versions" issue. They also help with the monorepo issue by encouraging developers publish the Nix derivation in their own repo.
But they currently have a bit of a discoverability issue, and they need adoption by legacy packages to completely fix the problem.
But they currently have a bit of a discoverability issue, and they need adoption by legacy packages to completely fix the problem.
I'll have to check out devbox, thanks for the link. I don't understand why Nix doesn't have some way for derivations to claim that they install certain artifacts, which would make searching so much easier.
The version thing is also really annoying, thanks for the helpful link, I hadn't seen that. Not sure how this will work in a Flakes world. Seems weird to me that sometimes packages are important enough to have multiple derivations each installing a different version (like the pythonFull packages) but there is no standard/good/default solution for it.
The version thing is also really annoying, thanks for the helpful link, I hadn't seen that. Not sure how this will work in a Flakes world. Seems weird to me that sometimes packages are important enough to have multiple derivations each installing a different version (like the pythonFull packages) but there is no standard/good/default solution for it.
Versioning is a bit easier with Flakes because they're generally published in their own Git repository, meaning you can use tags or branch refs to publish/install specific versions of a package. You can usually do something like `github:org/repo/ref#package` to get what you need.
Yes, but how will this work for the `nixpkgs` repository? Still very unclear!
I'm 100% onboard with flakes in general, and use flakes to describe devshells + publish binaries when applicable -- like with the tool I linked in my earlier post.
I'm 100% onboard with flakes in general, and use flakes to describe devshells + publish binaries when applicable -- like with the tool I linked in my earlier post.
> Finding a specific version of a package is also really difficult. It requires you to find the commit hash in the Nixpkgs repo that last contained it, and there isn't a central search engine that lets you map version to commit.
I don't understand this: Nixpkgs can only ever contain a tiny proportion of package versions. It's meant as a mostly-consistent set: e.g. if you want packages "foo", "bar" and "baz", that should work.
It's not intended for "foo-1.2", "bar-0.2" and "baz-99"; let alone toggling the super-duper compiler flag and applying Alice's stability patches. However, the latter are intended by the functions which define those packages; and it's exactly why so many "override" functions are provided (to do all of the above, we usually just need '.overrideAttrs').
If you want a consistent set of packages, the usual approach is to run some sort of dependency solver (e.g. many of the 'foo2nix' tools), and map over its result. For example, at work our Scala projects run 'mvn2nix' during their build; use import-from-derivation to turn the results into a local Maven repository; and use that to build the project in Maven's "offline mode".
I don't understand this: Nixpkgs can only ever contain a tiny proportion of package versions. It's meant as a mostly-consistent set: e.g. if you want packages "foo", "bar" and "baz", that should work.
It's not intended for "foo-1.2", "bar-0.2" and "baz-99"; let alone toggling the super-duper compiler flag and applying Alice's stability patches. However, the latter are intended by the functions which define those packages; and it's exactly why so many "override" functions are provided (to do all of the above, we usually just need '.overrideAttrs').
If you want a consistent set of packages, the usual approach is to run some sort of dependency solver (e.g. many of the 'foo2nix' tools), and map over its result. For example, at work our Scala projects run 'mvn2nix' during their build; use import-from-derivation to turn the results into a local Maven repository; and use that to build the project in Maven's "offline mode".
On the flipside, it's easy to change the version of any package to any git rev and automatically rebuild any downstream dependency.
or https://search.nixos.org
if folks want a easy (and good way) to get up and running in the nix ecosystem w/services ala replace docker compose then https://devenv.sh is the shizz
if folks want a easy (and good way) to get up and running in the nix ecosystem w/services ala replace docker compose then https://devenv.sh is the shizz
Also, package search is also prominently featured in the NixOS homepage for folks who don't want to remember the URL.
Ended up building a tool which indexes every version of every package into a searchable tool. Was hacked together in a weekend, so the UI needs some polishing, but it's functional :)
https://nixdex.dev
https://nixdex.dev
That's a use case, but for from the best. Nix lets you create immutable, reproducible packages/OSs/deploys--and can be built on arbitrary fleets of machines with shared caches. Just bringing in tools to the developer environment means the distribution of its tools is immutable but everything you plan to do with it inside that environment is mututable. The forest is missed for the trees if folks think dev shells are the killer feature.
devShells are nice because they have a really low investment to benefit ratio. They're nice because they can often be a clear win for many teams even if they have little prior Nix experience.
But you're right! Deeper investment, starting with actually performing your builds with Nix, yields greater benefits, and teams that never take that next step miss out on that.
But you're right! Deeper investment, starting with actually performing your builds with Nix, yields greater benefits, and teams that never take that next step miss out on that.
I wouldn’t disagree, but it’s like stepping into the kiddy pool without bothering to learn to swim to unlock all the other opportunities to swim (and possibly save others too).
> Speaks to the lack of user focus in the Nix community.
Not really. Nix is not a tool, it's a toolkit for making your own tools. Very much in the vein of Unix, shell, etc.
Many people don't use Nix for "installing" stuff. I personally use it for deploying software to servers without the (congnitive and performance) overhead of containers.
I find search.nixos.org immensely useful, but don't really need it from the CLI.
Not really. Nix is not a tool, it's a toolkit for making your own tools. Very much in the vein of Unix, shell, etc.
Many people don't use Nix for "installing" stuff. I personally use it for deploying software to servers without the (congnitive and performance) overhead of containers.
I find search.nixos.org immensely useful, but don't really need it from the CLI.
> Not really. Nix is not a tool, it's a toolkit for making your own tools. Very much in the vein of Unix, shell, etc
even tools for tools should be ergonomic to use. the Nix language server should have autocomplete, searches and interacting with the language should be easy, etc. and nearly everyone needs to hunt for packages, even if they're not "installing" them - they might be needed for the toolchain.
Nix is like early Git. all the raw power is there, but little of the porcelain. it's so much better than anything before it that we accept the pain of dealing with exposed plumbing, but we shouldn't get used to it.
even tools for tools should be ergonomic to use. the Nix language server should have autocomplete, searches and interacting with the language should be easy, etc. and nearly everyone needs to hunt for packages, even if they're not "installing" them - they might be needed for the toolchain.
Nix is like early Git. all the raw power is there, but little of the porcelain. it's so much better than anything before it that we accept the pain of dealing with exposed plumbing, but we shouldn't get used to it.
The last two years have seen an increased focus on usability, education and polish now that after circa 15 years of establishing solid foundations are in place. All good things in time.
There’s no venture backed cash here paying for the polish that folks typically see in devtools launched in 2023. That polish is expensive to do and isn’t “fun or engaging” for the majority of developers thus classic open source problem of over indexing on serving the needs/existing people who do not need the polish.
There’s some notable call-outs here of folks putting in serious effort such as https://nix.dev ala https://cachix.org out of love for nix to succeed.
Yeah nix wants to grow, and it is, largest bazaar GitHub community/repo out there right now.
starting to see venture backed companies forming who are building the polish as porcelain over the top and giving back. see https://floxdev.com/
at this stage nix is a damn safe bet because of what it can do, the problems it solves and the size of community.
There’s no venture backed cash here paying for the polish that folks typically see in devtools launched in 2023. That polish is expensive to do and isn’t “fun or engaging” for the majority of developers thus classic open source problem of over indexing on serving the needs/existing people who do not need the polish.
There’s some notable call-outs here of folks putting in serious effort such as https://nix.dev ala https://cachix.org out of love for nix to succeed.
Yeah nix wants to grow, and it is, largest bazaar GitHub community/repo out there right now.
starting to see venture backed companies forming who are building the polish as porcelain over the top and giving back. see https://floxdev.com/
at this stage nix is a damn safe bet because of what it can do, the problems it solves and the size of community.
> at this stage nix is a damn safe bet because of what it can do, the problems it solves and the size of community.
In the last few years, Nix has gone from a place where I was sometimes uncomfortable recommending it to people to one where I feel like it would be unstrategic not to use it where it fits. If you know Nix today, you should be pushing it at your company wherever you can see a good use case.
In the last few years, Nix has gone from a place where I was sometimes uncomfortable recommending it to people to one where I feel like it would be unstrategic not to use it where it fits. If you know Nix today, you should be pushing it at your company wherever you can see a good use case.
> In the last few years, Nix has gone from a place where I was sometimes uncomfortable recommending it to people to one where I feel like it would be unstrategic not to use it.
Super agree. Been using nix now for six(?) years and always been advocating for it but hesentiant to recommend companies adopt it (unless you can use it to attract talent as walmart did with nodejs back in the day) however with https://devenv.sh and https://nix.dev in the existence the time to amaze is now measured in < 5 minutes. nixpkgs also partially solves sbom topics, enables monkey patching any linux application/kernel/ffi dependency, has the freshest collection of packages (it's more fresh than archlinux by a major factor) whilst all being under CI/CD (shamefully rare in linux land)
Super agree. Been using nix now for six(?) years and always been advocating for it but hesentiant to recommend companies adopt it (unless you can use it to attract talent as walmart did with nodejs back in the day) however with https://devenv.sh and https://nix.dev in the existence the time to amaze is now measured in < 5 minutes. nixpkgs also partially solves sbom topics, enables monkey patching any linux application/kernel/ffi dependency, has the freshest collection of packages (it's more fresh than archlinux by a major factor) whilst all being under CI/CD (shamefully rare in linux land)
At flox, we are specifically focusing on those people who are the advocates and bringing Nix into their organizations, teams, and companies. We call them the “Ambassadors”, and want to call out how important that role is in making Nix mainstream. Ideally, it should become the obvious choice - the default choice - where specific reasons need to be provided when not using it.
I'm just saying that like Git and shell people will eventually make their own porcelain for a myriad different use cases. Most of those will probably be buried in some corporate proprietary repo.
Should? By whose standard?
Also nix does have some autocomplete, check out Nil…
Also nix does have some autocomplete, check out Nil…
What is confusing about finding Nix packages? Just go to search.nixos.org
There has been a trend in software dev towards solving problems that don’t need solved. I’ve never been confused about how to find packages for nix.
There has been a trend in software dev towards solving problems that don’t need solved. I’ve never been confused about how to find packages for nix.
It's genuinely nice to have a CLI frontend for search.nixos.org, so I wouldn't call it a problem that doesn't need solving. But the rant about not being able to search packages is simply wrong IMHO.
Or `nix search nixpkgs <query>`
If you try to run a binary from replit's shell it will find a package to run it with and will prompt you if there are multiple options
why does it query search.nixos.org, rather than looking through your local nixpkgs? just because of how tricky it'd be to handle the nix language or hacking on the nix toolchain proper?
also I agree that it's ridiculous nix doesn't have a search command. it baffles me.
also I agree that it's ridiculous nix doesn't have a search command. it baffles me.
This exists and searches `search.nixos.org` for a few reasons:
- you may want to search without downloading and indexing nixpkgs and the hydra results yourself
- you may want to search by the name of the binaries/programs that will be installed, which is impossible to do by looking through a local package store alone
- you may want the search to be fast
- you may want to search from the command line
`nix search` is a cruel joke which doesn't allow searching by the name/program that would be installed, only package name, and requires a flake name every time. Absolutely terrible interface.
My tool is a single-install binary that performs fast and accurate search to help you find the right package name to install a given binary. I don't understand how after years of using other package managers anyone could want a search tool that does anything other than this by default.
For more details on why this exists, check out https://github.com/peterldowns/nix-search-cli#motivation
- you may want to search without downloading and indexing nixpkgs and the hydra results yourself
- you may want to search by the name of the binaries/programs that will be installed, which is impossible to do by looking through a local package store alone
- you may want the search to be fast
- you may want to search from the command line
`nix search` is a cruel joke which doesn't allow searching by the name/program that would be installed, only package name, and requires a flake name every time. Absolutely terrible interface.
My tool is a single-install binary that performs fast and accurate search to help you find the right package name to install a given binary. I don't understand how after years of using other package managers anyone could want a search tool that does anything other than this by default.
For more details on why this exists, check out https://github.com/peterldowns/nix-search-cli#motivation
Nix has several search commands, it's just none of them do quite what people want (e.g. "nix search" with flakes and "nix-query" without flakes search by name and description, "nix-locate" can search by contents but the interface is a bit obtuse)
For those who want a local search for executables, and don't mind running nix-index:
For those who want a local search for executables, and don't mind running nix-index:
#!/bin/sh
results="$(nix-locate -1 --regex --top-level "^/bin/${1}\$")"
for item in $results; do
nix search "nixpkgs#${item%.*}"
done
search.nixos.org also includes flakes that people PR into the index via GitHub at https://github.com/NixOS/nixos-search/issues
ps: $ nix search exists via experimental flags https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3...
ps: $ nix search exists via experimental flags https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3...
It does though. `nix search`.
[deleted]
Dang this is cool! I get why replit went so heavy on nix but I also feel like it must have a cost for them — nix is hard to learn, especially for folks new to development which I know makes up a lot of replits customer base.
We built a solution to the same problem with a similar approach[1], but that just snapshots any old files instead of doing nix derivations. Nix couples the build process to the content-addressability of the output, which works great if you want to put all the effort in to deterministic builds. We just read files like git does which works great for non-deterministic processes like npm install (tragically).
I like the idea of the Big Disk style of attaching a content addressable cache, but in our experiments we still found the network latency to the attached disk too high when reading file by file, like when booting a node app, so we’re caching a much smaller amount on a local SSD for each prod server. Maybe replit isn’t as sensitive to read perf from the cache layer, or they have fancy local per-node read through caching within the overlay setup? Regardless, cool!!
[1]: https://github.com/gadget-inc/dateilager
We built a solution to the same problem with a similar approach[1], but that just snapshots any old files instead of doing nix derivations. Nix couples the build process to the content-addressability of the output, which works great if you want to put all the effort in to deterministic builds. We just read files like git does which works great for non-deterministic processes like npm install (tragically).
I like the idea of the Big Disk style of attaching a content addressable cache, but in our experiments we still found the network latency to the attached disk too high when reading file by file, like when booting a node app, so we’re caching a much smaller amount on a local SSD for each prod server. Maybe replit isn’t as sensitive to read perf from the cache layer, or they have fancy local per-node read through caching within the overlay setup? Regardless, cool!!
[1]: https://github.com/gadget-inc/dateilager
I love Nix as a package manager, and flakes are awesome for setting up environments. I wish it had more affordances for being a proper build system. If only there were a Buck2 and Nix hybrid.
> If only there were a Buck2 and Nix hybrid
You'll be interested in https://github.com/thoughtpolice/buck2-nix
You'll be interested in https://github.com/thoughtpolice/buck2-nix
bob.build looks kinda nice as a Nix-based, make-like tool, but IDK how it compares to the real, fancy, big tech monorepo build tools because I've never used them.
https://bob.build/
https://bob.build/
would love it if the compilation closure was more incremental rather than at a derivation level. specifically caching sub modules. many folks working on improving this space.
last week benchmarks from tvnix (a rust rewrite of the nix cli) was published which saw a THIRTEEN second evaluation speed up on a basic derivation evaluation.
https://twitter.com/matthewcroughan/status/16606138356553482...
last week benchmarks from tvnix (a rust rewrite of the nix cli) was published which saw a THIRTEEN second evaluation speed up on a basic derivation evaluation.
https://twitter.com/matthewcroughan/status/16606138356553482...
I saw that tweet and thought it was pretty exciting!
How much of Nixpkgs can Tvix build these days?
How much of Nixpkgs can Tvix build these days?
Those benchmarks are wrong and misleading. Tvix contributor (but mostly reviewer I would say) here. We still don't have finished the integration with our own store, using the Nix store forces us to be slower than Nix, therefore the claims are bogus alas.
Tvix can *evaluate* a serious (?) chunk of Nixpkgs, but we have strictness bugs in some areas.
Tvix has no builder yet.
Tvix can *evaluate* a serious (?) chunk of Nixpkgs, but we have strictness bugs in some areas.
Tvix has no builder yet.
Well, congrats to y'all on good progress with the evaluator, then! That's definitely still worth celebrating. :)
Nix Flakes are deemed experimental and explicitly not recommended in prod by the Nix project, afaict. Despite these warnings it’s widely used, but breakage is very much a possibility because it hasn’t gone through the Nix RFC process.
Where did anyone say "not recommended in prod"? The experimental tag is mostly saying they may change the API when they upgrade Nix itself.
Also, "in prod" is such an odd thing to say about Nix. It's a built-time thing. Nix doesn't run in prod. Even NixOS isn't running Nix code. It's running systemd units.
Also, "in prod" is such an odd thing to say about Nix. It's a built-time thing. Nix doesn't run in prod. Even NixOS isn't running Nix code. It's running systemd units.
> Also, "in prod" is such an odd thing to say about Nix. It's a built-time thing. Nix doesn't run in prod.
What people mean by "use X in production" is that X is a key dependency as part of the production process.
If you describe a NixOS configuration for the system where your services are running, you're "using" Nix, even though the Nix-specific stuff is already done.
This relates to risk. -- If it turned out that using Nix to build packages or to configure a system then caused problems when running services later, that would be a problem. Whereas, "using in prod" suggests confidence in no problems, or in being able to fix any issue in a timely manner.
What people mean by "use X in production" is that X is a key dependency as part of the production process.
If you describe a NixOS configuration for the system where your services are running, you're "using" Nix, even though the Nix-specific stuff is already done.
This relates to risk. -- If it turned out that using Nix to build packages or to configure a system then caused problems when running services later, that would be a problem. Whereas, "using in prod" suggests confidence in no problems, or in being able to fix any issue in a timely manner.
Stabilization of flakes and certain other experimental features is a top priority for the Nix team at the moment (disclaimer: myself included). Please take a look at RFC136 for an outline of our plan for the way ahead…… and please comment.
From the github issue [1]:
[1] - https://github.com/NixOS/rfcs/pull/136
First, the non-Flakes CLI wll be stabilized, in phases.
Afterwards, Flakes itself and its CLI components can be stabilized. The final design of Flakes will also require another RFC.
[1] - https://github.com/NixOS/rfcs/pull/136
Have been using flakes in production for more than 2 years - I think it is stable enough for daily use.
Flakes seem to be in "beta" the way Gmail was for years. Just about everyone I know that uses nix uses flakes now, and I've mostly switched myself.
I don't get how people find nix useful. Setting up a python environment, for example, is a total nightmare. Nix lang itself seems completely unnecessary. Guix is using scheme, which I assume people would be more incentivized to learn just because it's a general programming language. So why is nix winning?
I cannot comment on Guix versus Nix. I do agree the Nix language is unusual.
For instance, this statement:
The writer of the Nix expression does have to pay a significant cost, in my opinion. If done correctly, the users of the Nix expression can benefit enormously. Using a Nix project can be simple. These commands:
A very complex application written in multiple languages with multiple services can be set up with a Nix expression. The application developer then needs only the commands above to build, run, and test the application. Furthermore, Nix can provide the compiler and IDE support (such as an LSP server) to the developer.
For instance, this statement:
buildInputs = with pkgs;[
python34
python34Packages.docopt
];
The writer of the Nix expression does have to pay a significant cost, in my opinion. If done correctly, the users of the Nix expression can benefit enormously. Using a Nix project can be simple. These commands:
nix flake show
nix build ...
nix run ...
nix flake check
A very complex application written in multiple languages with multiple services can be set up with a Nix expression. The application developer then needs only the commands above to build, run, and test the application. Furthermore, Nix can provide the compiler and IDE support (such as an LSP server) to the developer.
So do they share their implementation of overlaying stores somewhere? It would be a huge benefit for the community to have it open source and available for being used in countless other use cases.
> In the coming weeks, we will be developing the Layered Store features, releasing a Nix Community RFC, and working to upstream it into Nix.
Doesn't look like specs or implementations are public yet.
Doesn't look like specs or implementations are public yet.
[deleted]
A less painful alternative to dancing around the issue is just to use the filesystem itself as a global database.
But oh my, the security implications. Nix is great, but there's no chain of trust on where these packages come from. Did we learn nothing from 'left_pad'?
Other than the very explicit tracking of the packages, their dependencies, the full build instructions, the public history, cryptographic signatures for the pre-built binaries, and the trivial ability for anyone to re-build from source and audit the entire chain all the way from bootstrap if they wished?
I’m not sure what golden standard we are comparing this to. It is not perfect, but I’d say this is a far more solid bedrock upon which to build software than anything else I’ve encountered.
I’m not sure what golden standard we are comparing this to. It is not perfect, but I’d say this is a far more solid bedrock upon which to build software than anything else I’ve encountered.
Did you mention sandboxed builds? Actively preventing pip to execute code arbitrarily at install time? Same for Node.js?
I cannot live without that anymore, that's the new golden standard.
I cannot live without that anymore, that's the new golden standard.
It’s funny; I forgot to mention sandboxed builds because at this point I’ve forgotten that they are not ubiquitous. Oops. Thanks!
> I’m not sure what golden standard we are comparing this to.
There isn't any, of course. But one high standard of excellence that we should celebrate and look up to (or just build upon!) is the Guix full-source bootstrap!
They've achieved something pretty incredible in terms of supply chain auditability: https://guix.gnu.org/blog/2023/the-full-source-bootstrap-bui...
Maybe we could also do more with Trustix, and/or build in something like `guix challenge` to the new Nix CLI, to make verifying package provenance and contents easier for users.
> I’d say this is a far more solid bedrock upon which to build software than anything else I’ve encountered.
With the exception (in some areas) of our little sister, Guix, absolutely. It's a real relief to have such a predictable, inspectable system as you get on NixOS. The drive for standards like software bills of materials comes from the chaos of the old world, the clumsy ways of building and distributing software, that proper functional package management like Nix obsoletes.
(Unfortunately we do still cope with and suffer from those old ways in Nixpkgs where the upstream build systems inflict it on us. The JVM and .NET come to mind.)
There isn't any, of course. But one high standard of excellence that we should celebrate and look up to (or just build upon!) is the Guix full-source bootstrap!
They've achieved something pretty incredible in terms of supply chain auditability: https://guix.gnu.org/blog/2023/the-full-source-bootstrap-bui...
Maybe we could also do more with Trustix, and/or build in something like `guix challenge` to the new Nix CLI, to make verifying package provenance and contents easier for users.
> I’d say this is a far more solid bedrock upon which to build software than anything else I’ve encountered.
With the exception (in some areas) of our little sister, Guix, absolutely. It's a real relief to have such a predictable, inspectable system as you get on NixOS. The drive for standards like software bills of materials comes from the chaos of the old world, the clumsy ways of building and distributing software, that proper functional package management like Nix obsoletes.
(Unfortunately we do still cope with and suffer from those old ways in Nixpkgs where the upstream build systems inflict it on us. The JVM and .NET come to mind.)
Heh, til. I'll have to take a closer look at the platform. Thank you!
There are no security implications other than those brought by the user. Freeze and host your own copies of libraries in an overlay (or write your own). The developer chooses which packages to use and from where, VERY explicitly and with a SHA which is far more secure than NPM for example (which is in production…everywhere)
Every nix package has to be signed in order for it to be installed via binary cache.
You can see exactly where everything's come from, using a command like:
nix-store --query --tree "$(nix-store --query --deriver "$(which python3)")"Nix is solving the wrong problems, and I'm dismayed that it is being adopted more and more to solve those wrong problems. The very concept of a "development environment" is indicative of broken build systems. Ecosystems like python are trying desperately to tame the insanity using venvs, while ecosystems like rust were built correctly from the start with Cargo. Neither of these benefits in the least way from Nix or the most popular alternative, Docker (except that both end up falling victim to the insanity of other ecosystems when FFI comes into play).
What Nix could have been was a global un-fscked dependency manager, a la Portage but with Nix's special sauce for local environments -- thereby working around fundamentally and permanently broken ecosystems like C, which would be useful even in the FFI case. But somehow it never got around to actually managing dependencies. By failing to manage dependencies (by pretending that versioning doesn't exist, and supporting only exact software sets), Nix only re-entrenches the same bad behavior and broken systems which it should have been replacing: i.e., encouraging "works-on-my-system" build environments, by making them part of mandatory collaboration workflows.
So, soon: Encounter a bug because the build depends on the environment in some inappropriate, insane way? Then it's your own fault for not adhering to the approved Nix build instructions which would set up a system which doesn't exhibit that bug.
What Nix could have been was a global un-fscked dependency manager, a la Portage but with Nix's special sauce for local environments -- thereby working around fundamentally and permanently broken ecosystems like C, which would be useful even in the FFI case. But somehow it never got around to actually managing dependencies. By failing to manage dependencies (by pretending that versioning doesn't exist, and supporting only exact software sets), Nix only re-entrenches the same bad behavior and broken systems which it should have been replacing: i.e., encouraging "works-on-my-system" build environments, by making them part of mandatory collaboration workflows.
So, soon: Encounter a bug because the build depends on the environment in some inappropriate, insane way? Then it's your own fault for not adhering to the approved Nix build instructions which would set up a system which doesn't exhibit that bug.
There is a lot to unpack here — first of all, Nix and Docker are not at all competitors. Docker doesn’t even try to be reproducible, most builds will just apt install the newest available version that could easily change between invocations.
Next, Cargo is not a solution: it can only do a tiny part of the problem, namely managing Rust dependencies for a Rust project.
What if you are building a tool that needs a shared lib to communicate with.. basically anything? Rust has an enthusiastic community, but it is still a tiny language — most problems require more than what can be found in any one language’s berks.
Also, what even is a version? 3.4 of package X doesn’t mean anything — if it has like 10 flags than it can be built 2^10 different ways, and then we didn’t even count its dependencies that are also not just a number. Nix does the correct thing, not the naive one.
Next, Cargo is not a solution: it can only do a tiny part of the problem, namely managing Rust dependencies for a Rust project.
What if you are building a tool that needs a shared lib to communicate with.. basically anything? Rust has an enthusiastic community, but it is still a tiny language — most problems require more than what can be found in any one language’s berks.
Also, what even is a version? 3.4 of package X doesn’t mean anything — if it has like 10 flags than it can be built 2^10 different ways, and then we didn’t even count its dependencies that are also not just a number. Nix does the correct thing, not the naive one.
Every single one of your paragraphs only makes sense in the context of ignoring my reference to portage.
Nix is not about "development environments" (i.e. 'nix-shell'); that's just a side-effect which comes for free. Nix's main entry-point is 'nix-build', which runs a specified executable with specified arguments and environment variables.
(The same applies for the newer 'nix' CLI, but I'm less familiar with that)
(The same applies for the newer 'nix' CLI, but I'm less familiar with that)
Is this a real problem? I've been using Nix for a year and a half constantly updating on 3 systems and haven't seen the issues you describe.
A "version" is an input like any other - treating it any differently would be a mistake.
I upvoted this even though I largely disagree with it, because I think it deserves discussion rather than downvotes. It's especially interesting to me because parts of it remind me of how I (along with many other Nix users) think of Docker.
> What Nix could have been was a global un-fscked dependency manager, a la Portage but with Nix's special sauce for local environments -- thereby working around fundamentally and permanently broken ecosystems like C, which would be useful even in the FFI case. But somehow it never got around to actually managing dependencies.
I agree. I think Nixpkgs can/should only officially support one combination of packages in a unified collection, up to the point of including pre-built binaries for that exact combination of dependencies in the main community binary caches.
But allowing multiple versions of packages to live in Nixpkgs (or an ancillary repo), automatically marked with version ranges reflecting the versions that those recipes have previously been successfully built against, would greatly simplify searching for and using packages by their upstream versions. The most trivial and most common case for overriding packages— changing the version of the sources to be built— would be obviated, leaving overrides to more advanced use cases like adding patches, changing build flags, adding dependencies, etc.
I think there's still room to grow Nix in this direction, though! The limitation here is incidental rather than fundamental.
> Nix only re-entrenches the same bad behavior and broken systems which it should have been replacing: i.e., encouraging "works-on-my-system" build environments, by making them part of mandatory collaboration workflows.
Not true. There is a problem that can occur where a lot of users or even collaborators on a project don't really know how to build it, or what all of the requirements are. This happens sometimes with projects where Docker is always the expected way to build or deploy. But sussing out the real build requirements from a Dockerfile is hard in a way that doing the same from a Nix package is not! Docker allows for all kinds of non-determinism, and dependencies in Docker can be supplied implicitly (or at least indirectly— you may have to track down how a particular base image was built to figure out if some dependency is assumed to be installed). But things like blocking network access and the prefix hacking in the Nix store ensure, if not guarantee, that the dependencies of Nix packages are explicit.
> So, soon: Encounter a bug because the build depends on the environment in some inappropriate, insane way? Then it's your own fault for not adhering to the approved Nix build instructions which would set up a system which doesn't exhibit that bug.
Nix is great, and it's great when projects offer a flake.nix or a shell.nix. But (1) development communities not knowing what their own requirements are is a cultural problem and (2) I hope and expect we'll see little of this. The kinds of things Nix does to environments that are Nix-specific are generally extremely unlikely to affect program behavior, like setting the date and build time to the start of the Unix epoch.
Have you encountered any projects that depend on Nix-specific behaviors? Or are you mainly worried that the kind of natural vendorization that Nix provides will lead developers to only test their software on narrower and narrower ranges of version combinations?
> What Nix could have been was a global un-fscked dependency manager, a la Portage but with Nix's special sauce for local environments -- thereby working around fundamentally and permanently broken ecosystems like C, which would be useful even in the FFI case. But somehow it never got around to actually managing dependencies.
I agree. I think Nixpkgs can/should only officially support one combination of packages in a unified collection, up to the point of including pre-built binaries for that exact combination of dependencies in the main community binary caches.
But allowing multiple versions of packages to live in Nixpkgs (or an ancillary repo), automatically marked with version ranges reflecting the versions that those recipes have previously been successfully built against, would greatly simplify searching for and using packages by their upstream versions. The most trivial and most common case for overriding packages— changing the version of the sources to be built— would be obviated, leaving overrides to more advanced use cases like adding patches, changing build flags, adding dependencies, etc.
I think there's still room to grow Nix in this direction, though! The limitation here is incidental rather than fundamental.
> Nix only re-entrenches the same bad behavior and broken systems which it should have been replacing: i.e., encouraging "works-on-my-system" build environments, by making them part of mandatory collaboration workflows.
Not true. There is a problem that can occur where a lot of users or even collaborators on a project don't really know how to build it, or what all of the requirements are. This happens sometimes with projects where Docker is always the expected way to build or deploy. But sussing out the real build requirements from a Dockerfile is hard in a way that doing the same from a Nix package is not! Docker allows for all kinds of non-determinism, and dependencies in Docker can be supplied implicitly (or at least indirectly— you may have to track down how a particular base image was built to figure out if some dependency is assumed to be installed). But things like blocking network access and the prefix hacking in the Nix store ensure, if not guarantee, that the dependencies of Nix packages are explicit.
> So, soon: Encounter a bug because the build depends on the environment in some inappropriate, insane way? Then it's your own fault for not adhering to the approved Nix build instructions which would set up a system which doesn't exhibit that bug.
Nix is great, and it's great when projects offer a flake.nix or a shell.nix. But (1) development communities not knowing what their own requirements are is a cultural problem and (2) I hope and expect we'll see little of this. The kinds of things Nix does to environments that are Nix-specific are generally extremely unlikely to affect program behavior, like setting the date and build time to the start of the Unix epoch.
Have you encountered any projects that depend on Nix-specific behaviors? Or are you mainly worried that the kind of natural vendorization that Nix provides will lead developers to only test their software on narrower and narrower ranges of version combinations?
> Or are you mainly worried that the kind of natural vendorization that Nix provides will lead developers to only test their software on narrower and narrower ranges of version combinations?
I guess, but I would not characterize it as a problem of testing. As I understand it the thing you are "supposed" to do is use the upstream developer's exact dependency tree, unless you override it in which case you've completely voided the warranty as it were. Or, this applies to collaborators across the same project too. This means using whatever broken outdated environment that developer happens to prefer, possibly because of other software on their system transitively requiring ridiculous environments, or because they're just lazy or whatever.
This doesn't apply to software in the blessed set that's kept up to date, but I don't think you get points for being at parity with, say, Ubuntu.
I guess, but I would not characterize it as a problem of testing. As I understand it the thing you are "supposed" to do is use the upstream developer's exact dependency tree, unless you override it in which case you've completely voided the warranty as it were. Or, this applies to collaborators across the same project too. This means using whatever broken outdated environment that developer happens to prefer, possibly because of other software on their system transitively requiring ridiculous environments, or because they're just lazy or whatever.
This doesn't apply to software in the blessed set that's kept up to date, but I don't think you get points for being at parity with, say, Ubuntu.
As someone who sometimes uses Nix to put together devShells for small projects at work, I admit there are times I might be inclined to point someone to the flake as a workaround for a versioning issue. But I would still consider it a bug in my app to be broken with a newer version of a library than what I have pinned. I believe many developers must feel similarly!
At the end of the day, when it comes to development environments, Nix should be a way to help provide a reproducible fast track for building and/or running the software, not a preemptive dismissal of all other possible downstream users and distros! If your software has a Nix flake but no build instructions, it's missing documentation. That's probably okay for solo open-source projects that are small and indifferent to building a userbase, but I hope maintainers and companies will readily recognize that as they grow they'll have to fill that gap at some point.
At the end of the day, when it comes to development environments, Nix should be a way to help provide a reproducible fast track for building and/or running the software, not a preemptive dismissal of all other possible downstream users and distros! If your software has a Nix flake but no build instructions, it's missing documentation. That's probably okay for solo open-source projects that are small and indifferent to building a userbase, but I hope maintainers and companies will readily recognize that as they grow they'll have to fill that gap at some point.
Not sure how if Nix/Replit has the program you’re looking for? Or what package name to use to actually get the program you want? Check out my cli search tool:
https://github.com/peterldowns/nix-search-cli
Edit: I can’t help myself from ranting, it’s insane that I had to write this tool. Speaks to the lack of user focus in the Nix community. Even the blog post here doesn’t mention how to actually find packages to install!
What good is a package manager that makes finding packages difficult? Who should be expected to tolerate such a thing?