Purely Functional Configuration Management with Nix and NixOS(infoq.com)
infoq.com
Purely Functional Configuration Management with Nix and NixOS
http://www.infoq.com/articles/configuration-management-with-nix
Configuration management is the foundation that makes modern infrastructure possible. Tools that enable configuration management are required in the toolbox of any operations team, and many development teams as well. Although all the tools aim to solve the same basic set of problems, they adhere to different visions and exhibit different characteristics. The issue is how to choose the tool that best fits each organization's scenarios.
7 comments
This is pretty wild! It looks like Nix, NixOS, & NixOps replace:
If the day to day details come anywhere close to the high level promise, this seems like it'll be totally amazing.
- low level package managers (eg rpm, dpkg)
- high level package managers (eg yum, apt)
- reproducible build tools
- configuration managers (eg puppet, chef)
- language-specific isolated env tools (eg rvm, virtualenv)
- language-specific package managers (eg bundler, pip)
- dev environment setup tools (eg vagrant)
- vm provisioning tools
- configuration source control (I think) (eg etckeeper)
- and probably more
If the day to day details come anywhere close to the high level promise, this seems like it'll be totally amazing.
It sounds almost too good to be true. The bottom line (as always):
* How hard is it to use?
* How easy is it to break?
* How hard is it to use?
* How easy is it to break?
It's easier to use than all of the above, since it just works nearly all of the time - the beauty of having identities for packages and complete dependency chains means that building software on one machine should be reproducible on another machine of the same architecture. Failure to build is not a human-error or missing dependency - it's a bug, and the package distributor is to blame, not the person trying to build.
It's difficult to break because packages are immutable, and the /nix/store where they are held is read-only - it can only be modified by a user with root privileges (which should be never), or the nix daemon (which runs as root).
The caveat is that packages have identities - you can't simply swap out a dependency for an equivalent version which has a different identity, without also updating all of the dependents of that package. "Swapping out" is the wrong term, since all versions can live alongside each other - it's more that, applications at the bottom of the chain will not automatically receive the updates of dependencies (for bug-fixes, etc) - you must also update all dependents to make use of such bug-fixes. For a bug-fix in the compiler for instance, that effectively means updating the entire distribution.
This could open up significant challenges in the area of live-patching on running systems, as the current research and tooling for this is built on the assumption that you can overwrite packages. On the other hand, I actually think Nix provides a better model for live-updating, because immutability is much more suited to making the kind of transactions we really need to perform them.
Arguably, shared libraries could be obsoleted and everything could be statically linked too, which could provide performance gains at the cost of memory/storage.
It's difficult to break because packages are immutable, and the /nix/store where they are held is read-only - it can only be modified by a user with root privileges (which should be never), or the nix daemon (which runs as root).
The caveat is that packages have identities - you can't simply swap out a dependency for an equivalent version which has a different identity, without also updating all of the dependents of that package. "Swapping out" is the wrong term, since all versions can live alongside each other - it's more that, applications at the bottom of the chain will not automatically receive the updates of dependencies (for bug-fixes, etc) - you must also update all dependents to make use of such bug-fixes. For a bug-fix in the compiler for instance, that effectively means updating the entire distribution.
This could open up significant challenges in the area of live-patching on running systems, as the current research and tooling for this is built on the assumption that you can overwrite packages. On the other hand, I actually think Nix provides a better model for live-updating, because immutability is much more suited to making the kind of transactions we really need to perform them.
Arguably, shared libraries could be obsoleted and everything could be statically linked too, which could provide performance gains at the cost of memory/storage.
It is pretty easy to use as a end user installing packages is no harder then other distributions. It does not have as wide of a selection as Debian, but I have found it easier to use their packaging system.
The packaging system is very friendly to beginners since it is straight forward to experiment with writing new packages installing them and then reverting if it does not work quite right. Or even better switching to a different profile with a different set of packages installed and coming back to your experiment later.
> How easy is it to break?
In the sense that the packages start conflicting with each other? The packages that come with nixpkgs rarely if every caused a conflict with other packages in nixpkgs. When they do I can simply roll back and make changes to the package.
If you explicitly mount the read only /nix/store as read/write and alter packages manually you can cause yourself some trouble, but it is something that is explicitly marked as dangerous and nearly unnecessary.
The packaging system is very friendly to beginners since it is straight forward to experiment with writing new packages installing them and then reverting if it does not work quite right. Or even better switching to a different profile with a different set of packages installed and coming back to your experiment later.
> How easy is it to break?
In the sense that the packages start conflicting with each other? The packages that come with nixpkgs rarely if every caused a conflict with other packages in nixpkgs. When they do I can simply roll back and make changes to the package.
If you explicitly mount the read only /nix/store as read/write and alter packages manually you can cause yourself some trouble, but it is something that is explicitly marked as dangerous and nearly unnecessary.
Back when I had debian on my home machine (a number of years ago), I experienced that dpkg would gradually lose track of the state of the system and become less and less useful over time. Once things got incoherent, there was no putting it back together, and it seemed inevitable after so many upgrades. I don't recall specifics, and I don't want to dwell on it, but just to illustrate how these things are predisposed to experience a kind of entropy death.
This one seems to be solving the right problems. But I hope they didn't get too carried away with solving the problem of being functional... that's not a real problem.
This one seems to be solving the right problems. But I hope they didn't get too carried away with solving the problem of being functional... that's not a real problem.
> I experienced that dpkg would gradually lose track of the state of the system and become less and less useful over time.
Similar experience with ubuntu and upgrading to different versions was my motivation to use nix. It was made worse by my efforts to install cutting edge versions of numpy and scipy at the time.
> But I hope they didn't get too carried away with solving the problem of being functional... that's not a real problem.
Heh, they are practical bunch, well at least good chunks of nix, nixos, hydra are written in perl and nixops is written in python. I think the main argument for the former was they all knew perl at the time and want to make sure it was reasonable to port to several systems including cygwin on windows thought it is no longer has tier one support.
Similar experience with ubuntu and upgrading to different versions was my motivation to use nix. It was made worse by my efforts to install cutting edge versions of numpy and scipy at the time.
> But I hope they didn't get too carried away with solving the problem of being functional... that's not a real problem.
Heh, they are practical bunch, well at least good chunks of nix, nixos, hydra are written in perl and nixops is written in python. I think the main argument for the former was they all knew perl at the time and want to make sure it was reasonable to port to several systems including cygwin on windows thought it is no longer has tier one support.
It would be nice if there was a github/remote integration. `nix-pkg -i github.com/my/project` would clone the repo and install the dependencies found in the deps.nix file in the root of the repo.
Right now, versions are just loosely selected so it would be (eg: there's a ruby193 and ruby20). If one version gets dropped in nigpkgs or the selector is not specific enough breakage would easily happen.
Right now, versions are just loosely selected so it would be (eg: there's a ruby193 and ruby20). If one version gets dropped in nigpkgs or the selector is not specific enough breakage would easily happen.
> nix-pkg -i github.com/my/project
What you describe would be fairly easy to script. I'm playing around with similar ideas. It's not exposed by default, but the included script `nix-prefetch-git` dumps a git repo into the nix store, and when you run it from your shell, you have access to ssh keys and ssl certs (that are unavailable within a normal nix build).
The next step is harder like you say, you'd either need to include nix expressions for all dependencies on down through libc and gcc, or just assume/hope that "whatever happens to be in the local nixpkgs" is appropriate, but then you kinda lose out on a lot of the point of nix. It'd be nice if there were some way to reference a version of nixpkgs from which the expressions for "ruby20" and its dependencies would be drawn (but then you'd end up with hundreds of slightly different versions of nixpkgs tarballs from every github project with no easy way to update).
What you describe would be fairly easy to script. I'm playing around with similar ideas. It's not exposed by default, but the included script `nix-prefetch-git` dumps a git repo into the nix store, and when you run it from your shell, you have access to ssh keys and ssl certs (that are unavailable within a normal nix build).
The next step is harder like you say, you'd either need to include nix expressions for all dependencies on down through libc and gcc, or just assume/hope that "whatever happens to be in the local nixpkgs" is appropriate, but then you kinda lose out on a lot of the point of nix. It'd be nice if there were some way to reference a version of nixpkgs from which the expressions for "ruby20" and its dependencies would be drawn (but then you'd end up with hundreds of slightly different versions of nixpkgs tarballs from every github project with no easy way to update).
Perhaps the easy way to update already exists - just that barely anyone uses it (properly). The feature is submodules - they provide exactly what we need for Nix - they give each commit a list of exact identities to commits in specific branches of submodules (ie, exact dependencies).
If we imagine a scenario where our bootstrap package already contains a .nixpkg in its root directory, and we augmented git with a pre-commit hook which mechanically extracted nixpkgs from all submodules, and we introduced a chain of git repositories where submodules are ubiquitously used for our entire system, maybe we could automatically produce the required .nixpkg for each commit (might require us to insert a "package template" into the root of each repository).
If we imagine a scenario where our bootstrap package already contains a .nixpkg in its root directory, and we augmented git with a pre-commit hook which mechanically extracted nixpkgs from all submodules, and we introduced a chain of git repositories where submodules are ubiquitously used for our entire system, maybe we could automatically produce the required .nixpkg for each commit (might require us to insert a "package template" into the root of each repository).
After some tire-kicking a few months ago, I've started using Nix in earnest, and I am loving it. It's sort of like moving from CVS to git: there's some new stuff to learn, but it's worth it.
I discovered Nix recently while searching for a _build_ tool that isn't limited by binary-only or source-only dependencies. If anyone is interested here is a collection of my thoughts/findings.
https://bitbucket.org/cofarrell/one-build-tool/src/master/RE...
Would love to hear any feedback people have about the idea.
https://bitbucket.org/cofarrell/one-build-tool/src/master/RE...
Would love to hear any feedback people have about the idea.
One of the benefits of updating software is to remove security vulnerabilities present in older versions.
Does use of Nix encourage insecurity by making it easy to use several outdated copies of each piece of software simultaneously?
Does use of Nix encourage insecurity by making it easy to use several outdated copies of each piece of software simultaneously?
No, because updating the vulnerable package will trigger rebuilds of the packages that depend on it. When the user upgrades, they will get the new, more secure version.
Agreed, though updating your system shouldn't break other packages. If those other packages are a security risk then they too should be updated. Once done, garbage collect that which is not used.
I've gotten over all the features of Nix, its become my environment of choice. What continuously impresses me now is the Nix community. An issue with your system can be resolved within minutes of a github issue submission. I'd like to see them use the ZeroMQ C4 development contract (http://rfc.zeromq.org/spec:16). No branches everywhere and it's just a rolling release. I highly recommend this distro, it has given me trouble in the growing stages but it's maturing nicely and things are starting to just work.
If I can ask, do you use this for a Linux Server you have, or your personal computer? I've been thinking about trying it out on my laptop, but wanted to know how well it does with graphics, networking and audio.
I've been an Ubuntu user for some years now, and feel pretty comfortable with the terminal, but haven't setup xwindows or ALSA manually or anything like that for example.
I've been an Ubuntu user for some years now, and feel pretty comfortable with the terminal, but haven't setup xwindows or ALSA manually or anything like that for example.
Okay when I first started using NixOS as a desktop, I knew it would bite me. But I'm a firm believer of the way NixOS handles state, so I suffered through both my and NixOS' growing pains. Basically I had to change my graphics card for some strange reason (which worked on Arch Linux!), ALSA gave me problems with flash videos, this is when I found out how amazing the NixOS community is. They helped me solve this issue quickly, it was a one liner in my configuration.nix. I've had issues with harddrives not being seen on time and having to tell the boot process to wait a second (again a one liner). It's actually quite a fun ride, don't install it on a machine you intend to do real work on, once happy and you've tweaked the hell out of your configuration.nix then roll it out where you need it. So if you have any issues, determine if you can resolve it intelligently and then create a github issue, they are very supportive (thank you Domen Kožar for the quick qt4 fix!). Now my system is happily ticking along, with one nagging Tk issue.
A very fair set of warnings. Thanks. I will keep thinking about it, and probably install it on my old laptop before doing anything serious with it on my desktop.
Check out GNU Guix, as well. https://gnu.org/software/guix/
It uses upstream Nix, but replaces the programming language with Scheme. Like NixOS, Guix is also a distribution: a fully free distribution of the GNU system.
It uses upstream Nix, but replaces the programming language with Scheme. Like NixOS, Guix is also a distribution: a fully free distribution of the GNU system.
(Please correct me if I'm wrong) but the NixOS functional language is also lazy. Guix does not have this lazy feature. Though on the other hand, it does play well with the rest of GNU.
Lazy evaluation in Scheme is achieved via delay/force macros, as well as simply quoting expressions to be evaluated later, perhaps by sending the quoted expression from the Guix client to the daemon.
Oh right, interesting. Now generally speaking I'd suspect things like concurrency, lazy and non-determinism should be made explicit in a language, these concepts should be language level not library/macro level. I'm quite curious to see how much this affects the expressiveness of the system. But I'm splitting hairs! This is LISP the old granddad of succinctness.