Show HN: FlakeHub Cache: Fast, secure, configurable. A new take on Nix caching(determinate.systems)
determinate.systems
Show HN: FlakeHub Cache: Fast, secure, configurable. A new take on Nix caching
https://determinate.systems/posts/flakehub-cache-beta/
5 comments
What worked great for me for speeding up CI was using Cachix + namespace.so. This works so well I moved all my personal and our company CI to GitHub Actions.
With "it works so well", I mean:
* CI is cheap because namespace.so runners are faster & cheaper than GitHub Actions
* the UX is exactly the same as using vanilla GitHub Actions, namespace.so's runner is a drop-in replacement
* the CI starts up very fast because `/nix/store` is stored on namespace.so's "Cached Volumes" which are like virtual disks attached to the CI runner, so there is nothing to download to prime the cache, Nix can just start running immediately
* https://github.com/cachix/cachix-action uploads whatever Nix builds to Cachix in the background so from this point on if any of our devs pulls the latest code, they don't have to build any derivations locally, and lose time waiting, everything is pulled automatically from Cachix to their machines, they can get right to work. Or if I have to purge the Cache Volume for some reason, nothing needs to be rebuilt, just pulled from Cachix.
It's taken a few years to get this dialed in, but now that it works, it's sooo good!
With "it works so well", I mean:
* CI is cheap because namespace.so runners are faster & cheaper than GitHub Actions
* the UX is exactly the same as using vanilla GitHub Actions, namespace.so's runner is a drop-in replacement
* the CI starts up very fast because `/nix/store` is stored on namespace.so's "Cached Volumes" which are like virtual disks attached to the CI runner, so there is nothing to download to prime the cache, Nix can just start running immediately
* https://github.com/cachix/cachix-action uploads whatever Nix builds to Cachix in the background so from this point on if any of our devs pulls the latest code, they don't have to build any derivations locally, and lose time waiting, everything is pulled automatically from Cachix to their machines, they can get right to work. Or if I have to purge the Cache Volume for some reason, nothing needs to be rebuilt, just pulled from Cachix.
It's taken a few years to get this dialed in, but now that it works, it's sooo good!
I'd be quite curious to see how using the Magic Nix Cache Action compares to your setup in terms of performance: https://github.com/DeterminateSystems/magic-nix-cache-action
(full disclosure: Determinate Systems employee)
(full disclosure: Determinate Systems employee)
IIUC Magic Nix Cache uses Github Actions Cache, which means there is waiting for the cache contents to be loaded onto the runner instance. This can in no way be as fast as what namespace.so does, which is mounting a disk image with /nix/store.
I'm going to have to look into this mounting option. I wanted to try making volumes very similar to this for reusing existing /target directories for internal company Rust builds. Sounds like Namespace could make this very easy to try out.
Just need to figure out what the cache size limitations are..
Just need to figure out what the cache size limitations are..
To be clear, I would not expect it to be as fast! But I do wonder what the percentage drop-off would be and, if small enough, if that would make staying on Actions more compelling than a bespoke solution.
What I've been doing for now is to pre-build the container to include the Nix environment in the nix store. I still run nix in the CI so I only have to rebuild the container from time to time, when the env has changed a lot. It's certainly not an ideal solution but it works without special support.
You indeed need some kind of nix-aware runner (e.g. with a persistent /nix) if you want both the flexibility and performance...
You indeed need some kind of nix-aware runner (e.g. with a persistent /nix) if you want both the flexibility and performance...
I am this close to building my own CI pipeline. Building a new CI pipeline is a really bad idea, I would be the first to tell you, and it's what I've been telling myself for five plus years now, but it's becoming harder to ignore the disparity between off the shelf tools that solve parts of the CI/CD problem and the ones that are actually used in garbage like Bamboo.
I think the fact that there is no bidding process for agents is a mistake that we need to correct. It's a little used concept in distributed computing, but a powerful one when applicable, and I think the preconditions for a fast build, with all of the docker images and other bootstrapping pipelines is making that cost harder to ignore. We are de facto using heterogeneous networks of worker processes now, and it's starting to reach 8 Fallacies level.
You build a distributed queue for response time or throughput, and we are using the throughput model, more or less, while CI/CD is mostly about response time.
I think the fact that there is no bidding process for agents is a mistake that we need to correct. It's a little used concept in distributed computing, but a powerful one when applicable, and I think the preconditions for a fast build, with all of the docker images and other bootstrapping pipelines is making that cost harder to ignore. We are de facto using heterogeneous networks of worker processes now, and it's starting to reach 8 Fallacies level.
You build a distributed queue for response time or throughput, and we are using the throughput model, more or less, while CI/CD is mostly about response time.
I don’t know about CircleCI but if you can persist the /nix/store then that will give you the performance improvements you’re looking for.
I use Gitlab CI and with self hosted runners I can make sure the /nix/store is persistent across CI jobs and then my builds take ~5 seconds if no code has changed, 5 seconds of which is just nix build evaluating all my nix files and realising no work needs to be done.
I use Gitlab CI and with self hosted runners I can make sure the /nix/store is persistent across CI jobs and then my builds take ~5 seconds if no code has changed, 5 seconds of which is just nix build evaluating all my nix files and realising no work needs to be done.
The problem people often run in to when persisting the nix store is that it can be very large, and updates often. This adds a significant tax on setup/teardown time and sometimes storage costs. The completely free and unsubsidized Magic Nix Cache's design alleviates that on GitHub Actions by persisting each path individually in the actions cache. FlakeHub Cache brings a for-cost component to that, which exposes the cache to your workstation and production infrastructure.
To clarify I meant having /nix/store on some kind of persistent volume rather than copying it at the start and end of every job, that indeed would be painfully slow and extremely inefficient. I don't know if this is possible with CircleCI.
Yes, this is exactly the problem I hit when I try to cache the Nix store on CircleCI.
This is part of why we built https://garnix.io/
Have you considered bundling that env into your CI image? I have a CI/CD process that will build and update an image with the environment pre-loaded. It doesn't save you the data transfer, but it likely provides a better warm cache.
I've thought about it, but I'm not sure how that would work. Because for me, the value proposition of having a Nix flake for my project is that I can bind the exact dependencies to the state of the code in my repo. So if I want to update from Go 1.19 to Go 1.20, I just update a line or two in my flake.nix. If I have to go update a separate repo, build a Docker image from that repo, then update the tag in my real repo, it kills the convenience.
Is there an easier way to do it?
Also, wouldn't it be pretty similar performance-wise to caching the /nix path on a generic Nix image? I've tried that, but it has problems because the Nix path ends up being a few GB, so that ends up adding a long time to the build as well.
Is there an easier way to do it?
Also, wouldn't it be pretty similar performance-wise to caching the /nix path on a generic Nix image? I've tried that, but it has problems because the Nix path ends up being a few GB, so that ends up adding a long time to the build as well.
It might help to describe how I employ this. I own the build and infra for a fairly large monorepo, and I moved all of the deps to Nix. I have a few ways to source/build things:
- A bare image with just Nix can pull the deps from a cache.
- A pre-built image contains the deps from the latest push to default branch and nothing else.
I do not maintain a separate repo for this. Part of my CI is building and caching the paths. And then CD is building and pushing the image, which the CI has verified as working. This requires using a mutable tag, but you could also have the CD process auto-update the tag if this is not an option.
You're correct that this doesn't save you from having to get the same amount of data to the same place, but the image repo can usually be cached more advantageously for whatever is pulling images, so it should be considerably faster overall.
- A bare image with just Nix can pull the deps from a cache.
- A pre-built image contains the deps from the latest push to default branch and nothing else.
I do not maintain a separate repo for this. Part of my CI is building and caching the paths. And then CD is building and pushing the image, which the CI has verified as working. This requires using a mutable tag, but you could also have the CD process auto-update the tag if this is not an option.
You're correct that this doesn't save you from having to get the same amount of data to the same place, but the image repo can usually be cached more advantageously for whatever is pulling images, so it should be considerably faster overall.
Also in the same boat.
I have the complication of android + Rust (com[compiling rust for android + SQLite require gimnastics: https://www.reddit.com/r/rust/comments/1872ftd/exist_a_solid...).
My ideal setup is to have my dev env be the CI env, to stop the chance that every so often things break (my CI breaks my delicate android incantions with some regularity).
I have the complication of android + Rust (com[compiling rust for android + SQLite require gimnastics: https://www.reddit.com/r/rust/comments/1872ftd/exist_a_solid...).
My ideal setup is to have my dev env be the CI env, to stop the chance that every so often things break (my CI breaks my delicate android incantions with some regularity).
I'd love to find out how it compares! We're not quite ready to support CircleCI, but when we are -- want to do a test? You could hop on to our discord: https://determinate.systems/discord
Hey folks, DetSys's cofounder here. Eelco, the team, and I have been working hard on FlakeHub Cache for some time now. We're starting to round out the toolkit you need to take Nix to work. Here to answer questions!
Beginner Nix question.
Regarding .dotfiles
It seems that there is a push or desire to have everything declared via nix and/or flakes. What I rarely see incorporated is 'git clone' or 'stow' for dotfile management. My simple question is: why not both? What is so wrong with using a 3rd party system as part of the deployment.
Let's say I want to fiddle with my vim/neovim .rc file. If I change it 10 times in the course of an hour, and run 'nixos rebuild switch' (did I get that right?) each time to set it, then I'm going to be spending a bunch of time/CPU cycles rebuilding world.
It seems to me a hybrid approach would work best. -nix rebuild scratch for OS-level modifications -git push/pull/clone for frequently modified files.
Nobody would ever suggest using the nix approach for storing their 'recipe folder'.
Regarding .dotfiles
It seems that there is a push or desire to have everything declared via nix and/or flakes. What I rarely see incorporated is 'git clone' or 'stow' for dotfile management. My simple question is: why not both? What is so wrong with using a 3rd party system as part of the deployment.
Let's say I want to fiddle with my vim/neovim .rc file. If I change it 10 times in the course of an hour, and run 'nixos rebuild switch' (did I get that right?) each time to set it, then I'm going to be spending a bunch of time/CPU cycles rebuilding world.
It seems to me a hybrid approach would work best. -nix rebuild scratch for OS-level modifications -git push/pull/clone for frequently modified files.
Nobody would ever suggest using the nix approach for storing their 'recipe folder'.
You don't actually need to rebuild the world, just the part that changed. It's rare to have other nix derivations depend on the dotfiles directly, so changes don't "cascade" to the rest of the system. I just timed how long it takes to rebuild my system after changing my neovim config and it's ~30 seconds. Which sounds like a lot compared to the zero seconds it takes to edit the config without nix, but it's fast enough that I personally don't mind.
Some people do use nix just for their system config and manage their dotfiles separately, but for me it's simpler to just have one system that installs the software I want and configures it the way I like it, rather than two things I'd need to keep in sync with each other.
Some people do use nix just for their system config and manage their dotfiles separately, but for me it's simpler to just have one system that installs the software I want and configures it the way I like it, rather than two things I'd need to keep in sync with each other.
Congrats on the product reveal! It's exciting to see all these things getting announced right around NixCon, and I wish I could have made it.
I'm having trouble seeing the audience for this product. If I'm in an org going for SOC 2, it's likely I already have a story around artifact access that isn't intrinsically tied with my build system. If I weren't already building sensitive things with Nix, this doesn't seem like the thing that would get me to switch. If I have a monorepo, flakes are likely off the table right away due to the performance hit. Am I missing something about how people are using flakes right now?
I'm having trouble seeing the audience for this product. If I'm in an org going for SOC 2, it's likely I already have a story around artifact access that isn't intrinsically tied with my build system. If I weren't already building sensitive things with Nix, this doesn't seem like the thing that would get me to switch. If I have a monorepo, flakes are likely off the table right away due to the performance hit. Am I missing something about how people are using flakes right now?
> Am I missing something about how people are using flakes right now?
Yes. Lots of orgs are using Nix today and need to meet SOC2 or better, and are using many different caches to meet compliance objectives. This is creating a significant performance penalty.
Further, whether you care about the separation or not the onboarding process is simply nicer, because it is tied to the environment you're running in (GHA) not tokens you have to copy paste around.
> If I have a monorepo, flakes are likely off the table right away due to the performance hit.
If you've heard of "lazy trees", that work is being spearheaded by us at DetSys to address this problem :).
Yes. Lots of orgs are using Nix today and need to meet SOC2 or better, and are using many different caches to meet compliance objectives. This is creating a significant performance penalty.
Further, whether you care about the separation or not the onboarding process is simply nicer, because it is tied to the environment you're running in (GHA) not tokens you have to copy paste around.
> If I have a monorepo, flakes are likely off the table right away due to the performance hit.
If you've heard of "lazy trees", that work is being spearheaded by us at DetSys to address this problem :).
Thanks for the response, Graham.
I think I'm still missing something. By all accounts, I and my organization should be in your target market, but there are a few barriers:
- I've achieved significant adoption of Nix for a great many things, but not for building proprietary software.
- I already have an enterprise-grade artifact store to use as a cache.
- I also have an identity platform and robust secrets management.
- I have evaluated lazy trees Nix at a few different intervals and did not find it enough of a speedup to continue using it.
The sum total of all of these barriers is that I may be wrong about whether we're truly in your target market, but I also think those will be common barriers for you, which makes me think I still don't quite understand the product or the market correctly.
I think I'm still missing something. By all accounts, I and my organization should be in your target market, but there are a few barriers:
- I've achieved significant adoption of Nix for a great many things, but not for building proprietary software.
- I already have an enterprise-grade artifact store to use as a cache.
- I also have an identity platform and robust secrets management.
- I have evaluated lazy trees Nix at a few different intervals and did not find it enough of a speedup to continue using it.
The sum total of all of these barriers is that I may be wrong about whether we're truly in your target market, but I also think those will be common barriers for you, which makes me think I still don't quite understand the product or the market correctly.
If you want to do caching but you aren't on Nix, check out Buildless:
https://less.build
https://less.build
One roadblock I haven't been able to overcome is integrating Nix in CI. In my case, I use CircleCI. I know there are Nix-centric CI platforms, but I've invested a lot into CircleCI and would like to stay.
The problem I hit is that if I switch from something like a sqlfluff Docker container to a Nix Docker container, my build job goes from 15s to 2m+ because now I have to download several gigs within the CI job and build my Nix environment from scratch.[1]
I tried Cachix, but the best I could get was bringing the job down to 55s (a roughly 3x performance penalty from an off-the-shelf Docker container).[2]
How does FlakeHub Cache's performance in CI compare to the default Nix cache or to Cachix?
[0] https://mtlynch.io/notes/nix-dev-environment/
[1] https://github.com/cachix/cachix/issues/579
[2] https://github.com/cachix/cachix/issues/579#issuecomment-175...