Structural and semantic deficiencies in the systemd architecture(blog.darknedgy.net)
blog.darknedgy.net
Structural and semantic deficiencies in the systemd architecture
http://blog.darknedgy.net/technology/2015/10/11/0/
11 comments
[deleted]
[deleted]
This is a very detailed article, and I appreciate it and will need to study it carefully.
But I think that the major argument it is making is "systemd is a mediocre service manager; similar tools existed before systemd, and were better." That is a very different argument from "we would be better off with sysvinit," which seems to be the primary argument I see in practice: if you're arguing things like "do one thing and do it well," "shell scripts are great and debuggable," "init should reap children and nothing else," "there's too much code in pid1," "I don't want to learn something new," etc., you're making that form of argument.
If there is general consensus from studied opponents of systemd that something systemd-like is an obviously good idea, great. Let's move on, acknowledge the network effects of systemd, and build something that can feasibly be the shinier, newer, better, securer systemd. Which might even be based on an existing system like s6. But then the path forward has to be either improving systemd or replacing it, not continuing with sysvinit.
I am in practice a pro-systemd person, but that's only because my realistic alternatives, for my use case, are sysvinit and Upstart. I have no idea how I'd start to build a system that used something like s6. I would happily admit there is extensive room for improvement in systemd, but we UNIX folks have been working with worse-is-better and rough-consensus-and-running-code for decades.
If the general consensus is that a systemd-style service manager is great, but it should not be pid 1, that's an argument worth making, but I don't see much focus in this article on that.
If the general consensus is that everything can be done with sysvinit, this article doesn't make that case at all. (But I suspect that's not the general consensus.)
But I think that the major argument it is making is "systemd is a mediocre service manager; similar tools existed before systemd, and were better." That is a very different argument from "we would be better off with sysvinit," which seems to be the primary argument I see in practice: if you're arguing things like "do one thing and do it well," "shell scripts are great and debuggable," "init should reap children and nothing else," "there's too much code in pid1," "I don't want to learn something new," etc., you're making that form of argument.
If there is general consensus from studied opponents of systemd that something systemd-like is an obviously good idea, great. Let's move on, acknowledge the network effects of systemd, and build something that can feasibly be the shinier, newer, better, securer systemd. Which might even be based on an existing system like s6. But then the path forward has to be either improving systemd or replacing it, not continuing with sysvinit.
I am in practice a pro-systemd person, but that's only because my realistic alternatives, for my use case, are sysvinit and Upstart. I have no idea how I'd start to build a system that used something like s6. I would happily admit there is extensive room for improvement in systemd, but we UNIX folks have been working with worse-is-better and rough-consensus-and-running-code for decades.
If the general consensus is that a systemd-style service manager is great, but it should not be pid 1, that's an argument worth making, but I don't see much focus in this article on that.
If the general consensus is that everything can be done with sysvinit, this article doesn't make that case at all. (But I suspect that's not the general consensus.)
Talking about "general concensus" is hard because there are a lot of different viewpoints, and depends on how much of the debate one is exposed to.
I have seen people compare systemd unfavourably to daemontools, s6, runit, Solaris' SMF, and Apple's launchd, to mention a few. I've never seen anyone so reactionary that they suggest that SysV is in any way or shape the better alternative. A lot of people are unhappy with SysV, but if they don't like systemd, they would probably consider sticking with SysV to be the lesser of two evils.
> Let's move on, acknowledge the network effects of systemd, and build something that can feasibly be the shinier, newer, better, securer systemd.
Unfortunately, I think that for those who don't like systemd, their criticisms (its design is a monolithic, intrusive, all-consuming, all-or-nothing ball of yarn contrary to Unix philosophy about small, composable programs cooperating over standard protocols) cannot be solved by improving systemd itself, at least not without a huge undertaking to redesign and break it apart, somehow.
I have seen people compare systemd unfavourably to daemontools, s6, runit, Solaris' SMF, and Apple's launchd, to mention a few. I've never seen anyone so reactionary that they suggest that SysV is in any way or shape the better alternative. A lot of people are unhappy with SysV, but if they don't like systemd, they would probably consider sticking with SysV to be the lesser of two evils.
> Let's move on, acknowledge the network effects of systemd, and build something that can feasibly be the shinier, newer, better, securer systemd.
Unfortunately, I think that for those who don't like systemd, their criticisms (its design is a monolithic, intrusive, all-consuming, all-or-nothing ball of yarn contrary to Unix philosophy about small, composable programs cooperating over standard protocols) cannot be solved by improving systemd itself, at least not without a huge undertaking to redesign and break it apart, somehow.
> I've never seen anyone so reactionary that they suggest that SysV is in any way or shape the better alternative.
Unfortunately it's... a position that is not as rare as it ought to be. Some of the frustration and heat undoubtedly comes from the fact that while reasonable people can disagree about the best way to implement many of the goals of systemd, it's hard to take anyone seriously when they propose that SysV init is "more Unix" or technically better.
> ...cannot be solved by improving systemd itself
I suspect the most fundamental impediment is that upstream simply aren't interested in radical changes to how it works - it would be like trying to convince Linus that Linux needs to be a microkernel, or Theo that he needs to start using Rust for OpenBSD.
Any replacement for systemd is probably going to have to do what systemd did (and what daemontools and runit didn't achieve) and create a broad buy-in, with the additional overhead of working with at least some of the systemd dbus interfaces for compatibility.
Unfortunately it's... a position that is not as rare as it ought to be. Some of the frustration and heat undoubtedly comes from the fact that while reasonable people can disagree about the best way to implement many of the goals of systemd, it's hard to take anyone seriously when they propose that SysV init is "more Unix" or technically better.
> ...cannot be solved by improving systemd itself
I suspect the most fundamental impediment is that upstream simply aren't interested in radical changes to how it works - it would be like trying to convince Linus that Linux needs to be a microkernel, or Theo that he needs to start using Rust for OpenBSD.
Any replacement for systemd is probably going to have to do what systemd did (and what daemontools and runit didn't achieve) and create a broad buy-in, with the additional overhead of working with at least some of the systemd dbus interfaces for compatibility.
It's important to realize that "a replacement for systemd" is neither achievable nor desirable.
systemd has grown fast and conquered large market shares because it was backed up by a company, which put a lot of money and manpower into it - to write it, to integrate it, to advertise it. The only thing that has not been correctly funded in systemd is research and design.
systemd is only Open Source if you read the license; all its other aspects are proprietary - it's software made by a company and aiming to capture a market. It is impossible to compete with systemd on the same grounds, because no real Open Source developers will have as many resources as Red Hat.
And even if it was possible, the result of such an attempt would simply be another integrated behemoth, powered by money and marketing instead of good technical decisions. (Or even worse, powered by ideology - can you imagine a systemd-like controlled by the FSF?) In the end, the situation for the users would be even worse than it is today. You don't fight Goliath with Goliath; you don't fight Microsoft's hegemony by buying Apple products.
About interface compatibility: the author of the DnE article (vezzy-fnord) has written uselessd, and finally abandoned the project because the systemd interfaces are so tightly integrated with the systemd design in the first place that it's impossible to be compatible without simply being a systemd clone, which he did not want uselessd to be. No, interface compatibility is not an option, because it would simply acknowledge the validity and superiority of the systemd architecture, and nobody wants a copy of systemd.
I believe that the way to provide an alternative to systemd is to provide all the functionality that the systemd users like, but in a technically better, less integrated, more unixish way.
With s6, s6-linux-init and s6-rc, I now have respectively a process supervision system, a simple init process and a service manager, which should be sufficient for a majority of applications. The next important thing that sysadmins like in systemd seems to be cgroup management, so I'd like to study the thing soon and assess what needs to be done next; but for now, I believe that the s6 family of programs is now viable as a serious alternative to systemd, and I would love to give it a broader audience.
systemd has grown fast and conquered large market shares because it was backed up by a company, which put a lot of money and manpower into it - to write it, to integrate it, to advertise it. The only thing that has not been correctly funded in systemd is research and design.
systemd is only Open Source if you read the license; all its other aspects are proprietary - it's software made by a company and aiming to capture a market. It is impossible to compete with systemd on the same grounds, because no real Open Source developers will have as many resources as Red Hat.
And even if it was possible, the result of such an attempt would simply be another integrated behemoth, powered by money and marketing instead of good technical decisions. (Or even worse, powered by ideology - can you imagine a systemd-like controlled by the FSF?) In the end, the situation for the users would be even worse than it is today. You don't fight Goliath with Goliath; you don't fight Microsoft's hegemony by buying Apple products.
About interface compatibility: the author of the DnE article (vezzy-fnord) has written uselessd, and finally abandoned the project because the systemd interfaces are so tightly integrated with the systemd design in the first place that it's impossible to be compatible without simply being a systemd clone, which he did not want uselessd to be. No, interface compatibility is not an option, because it would simply acknowledge the validity and superiority of the systemd architecture, and nobody wants a copy of systemd.
I believe that the way to provide an alternative to systemd is to provide all the functionality that the systemd users like, but in a technically better, less integrated, more unixish way.
With s6, s6-linux-init and s6-rc, I now have respectively a process supervision system, a simple init process and a service manager, which should be sufficient for a majority of applications. The next important thing that sysadmins like in systemd seems to be cgroup management, so I'd like to study the thing soon and assess what needs to be done next; but for now, I believe that the s6 family of programs is now viable as a serious alternative to systemd, and I would love to give it a broader audience.
> systemd has grown fast and conquered large market shares because it was backed up by a company, which put a lot of money and manpower into it - to write it, to integrate it, to advertise it. The only thing that has not been correctly funded in systemd is research and design.
> systemd is only Open Source if you read the license; all its other aspects are proprietary - it's software made by a company and aiming to capture a market. It is impossible to compete with systemd on the same grounds, because no real Open Source developers will have as many resources as Red Hat.
This. The reason systemd is hated so much by "old unix guys" is because it shows that Red Hat basically owns linux the way Microsoft owns Windows and Apple owns OS X.
Many people protested against systemd but it was still pushed by force. It was made specificaly incompatible to make experience on BSDs worse and it sucessfuly removed Gnome from there.
So yeah. Some "haters" talk about unix this and complex that, but I believe it's mostly about scummy company showing off it owns linux now.
> systemd is only Open Source if you read the license; all its other aspects are proprietary - it's software made by a company and aiming to capture a market. It is impossible to compete with systemd on the same grounds, because no real Open Source developers will have as many resources as Red Hat.
This. The reason systemd is hated so much by "old unix guys" is because it shows that Red Hat basically owns linux the way Microsoft owns Windows and Apple owns OS X.
Many people protested against systemd but it was still pushed by force. It was made specificaly incompatible to make experience on BSDs worse and it sucessfuly removed Gnome from there.
So yeah. Some "haters" talk about unix this and complex that, but I believe it's mostly about scummy company showing off it owns linux now.
I would say it's more like RedHat has actually stepped up and made a "Linux-derived OS" the way that Android is a "Linux-derived OS" or OSX is a "BSD-derived OS." They've thrown the "basic Unix principles"—the ones that make Linux distributions effectively interchangable commodities, but with software having to target a very low lowest-common-denominator of functionality—to the side, and instead adopted a complete "base system API", in the same sense that CoreFoundation+Cocoa is for OSX, and Win32 is for Windows.
As with those other APIs, you can now "target systemd" in the same way you could "target Win32" or "target Android"—and, because of this, we'll probably see a virtualization library (like winelib is for Win32; like ARC is for Android) spring up, giving other OSes the ability to provide the "systemd API" without needing to run a Linux kernel.
From RedHat's perspective, they were just trying to copy OSX and Android: making a "real OS" atop Unix-y foundations. The interesting thing is that a large number of other providers in the Linux space has started to agree with their design decisions—effectively unifying on the "systemd platform" rather than on the "Posix platform" they were focusing on previously.
I think this has been coming for a long time; the large OS providers already think of Desktop Environments as the GUI equivalent to a "platform" rather than as application software for the OS; thus why Ubuntu, for example, has a separate distribution for each GUI software suite.
In a discussion of systemd history held here previously, it was pointed out that back in 2007-or-so, there was a large amount of interest expressed by various distro makers in adopting OSX's launchd as the basis for a new platform standard, if only it had been FOSS at the time. Early systemd was a launchd clone; upstart also started as a launchd clone before mutating.
I have a feeling that although RedHat was the one to step up and create a platform, all of the major distro providers were actually waiting to hop on the first available FOSS "OS platform", no matter who it came from. If, for example, Apple had created "Apple Linux": a version of Linux that runs the OSX GUI and OSX apps—then I can bet you right now that every other distro would have standardized on whatever "platform" gunk was in-between that GUI and the Linux kernel; CoreFoundation et al would have become to Linux distros as WebKit is to mobile web browsers.
As it stands, it's only an accident of history that this didn't happen with the Android platform. (If the Android Runtime ran native-speed binaries from the start, and the GUI was developed five-or-so years later such that it was developed from the start with Win8-like unified desktop/mobile support in mind rather than getting type-cast as "only for mobile devices", I could bet that Linux would "be" Android now.)
---
With all that said, while all the big, commercially-backed Linux providers have been waiting on tenterhooks for a "platform" to latch onto, all the little distros are looking to remain Unix-y, and that's a wonderful thing.
Think about it: when you've got the POSIX platform "for desktops and servers" and the Android platform "for mobile", and nary the twain shall meet, it makes little sense to care about cross-compatibility; why would you want to run Android apps natively on your desktop? But if you add in a third platform that's also for desktop—the systemd platform—suddenly kernel developers and distro makers and whatever you call the FreeDesktop people start thinking about how to create a Linux-based OS that natively supports multiple platforms, so that it can run code targeting systemd, and code targeting Plain-old-POSIX, without making a mess. And once you have that infrastructure in place... why not let it run code targeting Android too? Why not first-class Wine support? Etc.
One thing Windows has had forever (though lately in a degraded state) is multi-runtime (or, given the above discourse, multi-"platform") support. Windows can be Win32 to one program, OS/2 to another, and POSIX to a third. OSX had the capability for multi-runtime support back when it was called Rhapsody (the "Red Box, Blue Box, Yellow Box" architecture) and also co-supported Cocoa and Carbon apps for about a decade. And these are just single companies trying to meet small needs. Linux, the OS with a kernel that supports every device a nerd ever cared to packet-probe, could easily become "the multi-runtime OS", if we care enough to make it that way.
If we don't, though, at least systemd is decent. :P
As with those other APIs, you can now "target systemd" in the same way you could "target Win32" or "target Android"—and, because of this, we'll probably see a virtualization library (like winelib is for Win32; like ARC is for Android) spring up, giving other OSes the ability to provide the "systemd API" without needing to run a Linux kernel.
From RedHat's perspective, they were just trying to copy OSX and Android: making a "real OS" atop Unix-y foundations. The interesting thing is that a large number of other providers in the Linux space has started to agree with their design decisions—effectively unifying on the "systemd platform" rather than on the "Posix platform" they were focusing on previously.
I think this has been coming for a long time; the large OS providers already think of Desktop Environments as the GUI equivalent to a "platform" rather than as application software for the OS; thus why Ubuntu, for example, has a separate distribution for each GUI software suite.
In a discussion of systemd history held here previously, it was pointed out that back in 2007-or-so, there was a large amount of interest expressed by various distro makers in adopting OSX's launchd as the basis for a new platform standard, if only it had been FOSS at the time. Early systemd was a launchd clone; upstart also started as a launchd clone before mutating.
I have a feeling that although RedHat was the one to step up and create a platform, all of the major distro providers were actually waiting to hop on the first available FOSS "OS platform", no matter who it came from. If, for example, Apple had created "Apple Linux": a version of Linux that runs the OSX GUI and OSX apps—then I can bet you right now that every other distro would have standardized on whatever "platform" gunk was in-between that GUI and the Linux kernel; CoreFoundation et al would have become to Linux distros as WebKit is to mobile web browsers.
As it stands, it's only an accident of history that this didn't happen with the Android platform. (If the Android Runtime ran native-speed binaries from the start, and the GUI was developed five-or-so years later such that it was developed from the start with Win8-like unified desktop/mobile support in mind rather than getting type-cast as "only for mobile devices", I could bet that Linux would "be" Android now.)
---
With all that said, while all the big, commercially-backed Linux providers have been waiting on tenterhooks for a "platform" to latch onto, all the little distros are looking to remain Unix-y, and that's a wonderful thing.
Think about it: when you've got the POSIX platform "for desktops and servers" and the Android platform "for mobile", and nary the twain shall meet, it makes little sense to care about cross-compatibility; why would you want to run Android apps natively on your desktop? But if you add in a third platform that's also for desktop—the systemd platform—suddenly kernel developers and distro makers and whatever you call the FreeDesktop people start thinking about how to create a Linux-based OS that natively supports multiple platforms, so that it can run code targeting systemd, and code targeting Plain-old-POSIX, without making a mess. And once you have that infrastructure in place... why not let it run code targeting Android too? Why not first-class Wine support? Etc.
One thing Windows has had forever (though lately in a degraded state) is multi-runtime (or, given the above discourse, multi-"platform") support. Windows can be Win32 to one program, OS/2 to another, and POSIX to a third. OSX had the capability for multi-runtime support back when it was called Rhapsody (the "Red Box, Blue Box, Yellow Box" architecture) and also co-supported Cocoa and Carbon apps for about a decade. And these are just single companies trying to meet small needs. Linux, the OS with a kernel that supports every device a nerd ever cared to packet-probe, could easily become "the multi-runtime OS", if we care enough to make it that way.
If we don't, though, at least systemd is decent. :P
In a discussion of systemd history held here previously, it was pointed out that back in 2007-or-so, there was a large amount of interest expressed by various distro makers in adopting OSX's launchd as the basis for a new platform standard, if only it had been FOSS at the time.
I don't think there ever was a large amount of interest. In fact, the only party that studied launchd at the time (circa 2006) was Canonical. It was indeed licensing issues that stalled further research, but there was also another contender at the time being considered called initng, which Canonical ended up rejecting and went on to write Upstart instead, led by Scott James Remnant. See SJR's proposal and introduction to Upstart. [1]
Early systemd was a launchd clone; upstart also started as a launchd clone before mutating.
Upstart was never a launchd clone to the best of my knowledge. launchd was likely a spark that influenced Canonical to take action, but the design is pretty different.
Your musings about the "multi-runtime" convergence that systemd will allegedly enable do not appear to pan out. systemd is nothing like the OS X Core frameworks or like the Windows Runtime, it's much lower level than that. It's more of a middleware than a runtime platform (think Hurd, not Core Foundation).
There is also absolutely nothing implying that GNU/Linux will ever target Android code. The divergences are plentiful, with my article about an Android init porting attempt listing only a few. [2] Nor is there any expressed interest from any Android vendor to have any serious GNU/Linux convergence to begin with.
Not sure what "first-class Wine support" is supposed to mean. Wine is pretty self-contained.
If we don't, though, at least systemd is decent.
You're too big of a dreamer, I'm afraid.
[1] https://wiki.ubuntu.com/ReplacementInit
[2] http://blog.darknedgy.net/technology/2015/08/05/0-androidini...
I don't think there ever was a large amount of interest. In fact, the only party that studied launchd at the time (circa 2006) was Canonical. It was indeed licensing issues that stalled further research, but there was also another contender at the time being considered called initng, which Canonical ended up rejecting and went on to write Upstart instead, led by Scott James Remnant. See SJR's proposal and introduction to Upstart. [1]
Early systemd was a launchd clone; upstart also started as a launchd clone before mutating.
Upstart was never a launchd clone to the best of my knowledge. launchd was likely a spark that influenced Canonical to take action, but the design is pretty different.
Your musings about the "multi-runtime" convergence that systemd will allegedly enable do not appear to pan out. systemd is nothing like the OS X Core frameworks or like the Windows Runtime, it's much lower level than that. It's more of a middleware than a runtime platform (think Hurd, not Core Foundation).
There is also absolutely nothing implying that GNU/Linux will ever target Android code. The divergences are plentiful, with my article about an Android init porting attempt listing only a few. [2] Nor is there any expressed interest from any Android vendor to have any serious GNU/Linux convergence to begin with.
Not sure what "first-class Wine support" is supposed to mean. Wine is pretty self-contained.
If we don't, though, at least systemd is decent.
You're too big of a dreamer, I'm afraid.
[1] https://wiki.ubuntu.com/ReplacementInit
[2] http://blog.darknedgy.net/technology/2015/08/05/0-androidini...
> systemd is nothing like the OS X Core frameworks or like the Windows Runtime, it's much lower level than that. It's more of a middleware than a runtime platform
You're not considering the full scope of the thing labelled "systemd." If you only use parts of it, it's middleware, yes. If you "work with the grain" of systemd, though, then you're packaging services in nspawn containers and so forth, which does constitute a separate "platform target", in the same way that e.g. CoreOS is a (mostly-ignored attempt at a) "platform target."
Basically, I'm talking here about a Linux equivalent to Solaris's "Branded Zones": within a container boundary, an app can be made by the system to think it's running on "systemd Linux", or on "POSIX Linux", or on "Android Linux", or on BSD or Win32 or Cocoa or whatever else. Runtime-virtualization is done at the system container-management level (with more or less help from the kernel), rather than expecting applications to "port" themselves by applying their own proprietary virtualization wrappers ala Cider for OSX.
> Not sure what "first-class Wine support" is supposed to mean. Wine is pretty self-contained.
I mean, basically, support to the level of DOS applications run in Win32 VMM containers, rather than to the level of X11 applications run on OSX: management of Wine sandboxes as OS-level "runtime containers", such that you could run and maintain Wine apps alongside other apps, in production, using the OS's maintenance tooling. A Linux "ReactOS Runtime" equivalent to Windows's "POSIX Runtime."
> Nor is there any expressed interest from any Android vendor to have any serious GNU/Linux convergence to begin with.
I don't mean to suggest convergence. This is something very different: making "running an Android binary" a primitive action of the system (not the Linux kernel), where "using an Android virtualization layer" is then an implementation detail of how the system accomplishes this. Think of Linux's binfmt_misc and its ability to execute e.g. JVM code by booting a JVM, but with an upcall to the init daemon to decide how to implement the policy of running a particular binary format. If that init daemon is effectively a container-manager that understands how to instantiate and manage different "branded containers" for each runtime it supports, Linux multi-runtime support just falls out.
You're not considering the full scope of the thing labelled "systemd." If you only use parts of it, it's middleware, yes. If you "work with the grain" of systemd, though, then you're packaging services in nspawn containers and so forth, which does constitute a separate "platform target", in the same way that e.g. CoreOS is a (mostly-ignored attempt at a) "platform target."
Basically, I'm talking here about a Linux equivalent to Solaris's "Branded Zones": within a container boundary, an app can be made by the system to think it's running on "systemd Linux", or on "POSIX Linux", or on "Android Linux", or on BSD or Win32 or Cocoa or whatever else. Runtime-virtualization is done at the system container-management level (with more or less help from the kernel), rather than expecting applications to "port" themselves by applying their own proprietary virtualization wrappers ala Cider for OSX.
> Not sure what "first-class Wine support" is supposed to mean. Wine is pretty self-contained.
I mean, basically, support to the level of DOS applications run in Win32 VMM containers, rather than to the level of X11 applications run on OSX: management of Wine sandboxes as OS-level "runtime containers", such that you could run and maintain Wine apps alongside other apps, in production, using the OS's maintenance tooling. A Linux "ReactOS Runtime" equivalent to Windows's "POSIX Runtime."
> Nor is there any expressed interest from any Android vendor to have any serious GNU/Linux convergence to begin with.
I don't mean to suggest convergence. This is something very different: making "running an Android binary" a primitive action of the system (not the Linux kernel), where "using an Android virtualization layer" is then an implementation detail of how the system accomplishes this. Think of Linux's binfmt_misc and its ability to execute e.g. JVM code by booting a JVM, but with an upcall to the init daemon to decide how to implement the policy of running a particular binary format. If that init daemon is effectively a container-manager that understands how to instantiate and manage different "branded containers" for each runtime it supports, Linux multi-runtime support just falls out.
nspawn containers are yet not really used as a platform target, and currently Docker is leading. Though nspawn has since acquired some notion of the Docker image format, and even in light of the OCI standard, I do not foresee it becoming a primary container solution in its present form. rkt being nspawn-based may or may not take hold.
Indeed, systemd is as of yet not comprehensive enough to be a POSIX-parity target. There is no "systemd Linux" as such, it's systemd/GNU/Linux or Linux with the GNU and systemd suite. Android is a top-down integrated system on the other hand and does not linearly track GNU/Linux.
As of yet, there is nothing like system call emulation or similar in nspawn to have branded zones.
Nor does Red Hat's present actions imply something like this. The GNOME project, affiliated with Red Hat, is working on a poor reinvention of Nix called xdg-app to enable the "app frameworks and runtimes" design that Lennart Poettering wrote about in "Revisiting How We Put Together Linux Systems," but that too is firmly specific to GNU/Linux as the host.
Red Hat is also leading a container OS called Project Atomic, however nothing like branded zones is seen there as a goal, either. Instead, they've made a simple meta-framework for running various Linux container images over several orchestration frameworks and PaaS, called Nulecule. It's firmly a layer over Docker and Kubernetes, however, so it is limited to that.
A Linux "ReactOS Runtime" equivalent to Windows's "POSIX Runtime."
That would be quite a feat in of itself, systemd or branded zones aside. ReactOS isn't Wine, but even with Wine it would be a sizable integration effort.
If that init daemon is effectively a container-manager that understands how to instantiate and manage different "branded containers" for each runtime it supports, Linux multi-runtime support just falls out.
The init daemon is not a container manager in this case, but an object-oriented resource management with a transactional job scheduler and some limited execution environment modification that work with namespaces and cgroups, but the container framework is outside. As it should be. An init daemon as your container manager sounds dreadful and horrifying, though I hear RancherOS boots from Docker as PID1...
But as there is no such init daemon yet nor anything like branded zones, I still have to say you're a dreamer. This might be a long-term strategy, but with acts like Project Atomic it really doesn't look like it. I still think it's more middleware than Core runtime. I pray it is...
Indeed, systemd is as of yet not comprehensive enough to be a POSIX-parity target. There is no "systemd Linux" as such, it's systemd/GNU/Linux or Linux with the GNU and systemd suite. Android is a top-down integrated system on the other hand and does not linearly track GNU/Linux.
As of yet, there is nothing like system call emulation or similar in nspawn to have branded zones.
Nor does Red Hat's present actions imply something like this. The GNOME project, affiliated with Red Hat, is working on a poor reinvention of Nix called xdg-app to enable the "app frameworks and runtimes" design that Lennart Poettering wrote about in "Revisiting How We Put Together Linux Systems," but that too is firmly specific to GNU/Linux as the host.
Red Hat is also leading a container OS called Project Atomic, however nothing like branded zones is seen there as a goal, either. Instead, they've made a simple meta-framework for running various Linux container images over several orchestration frameworks and PaaS, called Nulecule. It's firmly a layer over Docker and Kubernetes, however, so it is limited to that.
A Linux "ReactOS Runtime" equivalent to Windows's "POSIX Runtime."
That would be quite a feat in of itself, systemd or branded zones aside. ReactOS isn't Wine, but even with Wine it would be a sizable integration effort.
If that init daemon is effectively a container-manager that understands how to instantiate and manage different "branded containers" for each runtime it supports, Linux multi-runtime support just falls out.
The init daemon is not a container manager in this case, but an object-oriented resource management with a transactional job scheduler and some limited execution environment modification that work with namespaces and cgroups, but the container framework is outside. As it should be. An init daemon as your container manager sounds dreadful and horrifying, though I hear RancherOS boots from Docker as PID1...
But as there is no such init daemon yet nor anything like branded zones, I still have to say you're a dreamer. This might be a long-term strategy, but with acts like Project Atomic it really doesn't look like it. I still think it's more middleware than Core runtime. I pray it is...
Note that I wasn't suggesting that systemd is itself an attempt at an "init daemon with branded zones"—instead, it's a particular runtime that will increasingly differentiate itself from POSIX Linux, and I believe that that friction will eventually cause developers to want to create an init daemon with branded zones to supercede systemd, where emulating systemd-as-it-stands (for apps built to expect it) would be one of that multi-runtime system's goals. With the POSIX people and the bigcorp Linux providers pulling in opposite directions, a multi-runtime Linux would be the only Nash equilibrium. (Though it'd need someone like FreeDesktop to suggest it and start a working group for it, because neither "side" would care about it on their own.)
That may not be the way things play out, certainly. Things might just diverge and stay that way, if nobody cares enough to change things. But there are unknown unknowns that can give things a very hard shove in that direction. For example: imagine that systemd decides to integrate deeply with GNOME, to the point that you now have an integrated "systemd+glib+GNOME platform" with a unified API, the way that OSX is a "launchd+CoreFoundation+Cocoa platform". That would be an extremely divergent target in the Linux ecosystem introducing a lot of friction to everyone else's development efforts—similar to the early stages of the Android project—and it would get a lot of people's hackles up. But it's not out of the realm of possibility for RedHat and Debian to agree on something like that.
That may not be the way things play out, certainly. Things might just diverge and stay that way, if nobody cares enough to change things. But there are unknown unknowns that can give things a very hard shove in that direction. For example: imagine that systemd decides to integrate deeply with GNOME, to the point that you now have an integrated "systemd+glib+GNOME platform" with a unified API, the way that OSX is a "launchd+CoreFoundation+Cocoa platform". That would be an extremely divergent target in the Linux ecosystem introducing a lot of friction to everyone else's development efforts—similar to the early stages of the Android project—and it would get a lot of people's hackles up. But it's not out of the realm of possibility for RedHat and Debian to agree on something like that.
instead, it's a particular runtime that will increasingly differentiate itself from POSIX Linux
Ah, correct.
I believe that that friction will eventually cause developers to want to create an init daemon with branded zones to supercede systemd, where emulating systemd-as-it-stands (for apps built to expect it) would be one of that multi-runtime system's goals. With the POSIX people and the bigcorp Linux providers pulling in opposite directions, a multi-runtime Linux would be the only Nash equilibrium. (Though it'd need someone like FreeDesktop to suggest it and start a working group for it, because neither "side" would care about it on their own.)
It's a hypothesis, then. Plausible, certainly. I assume you're presenting this as a probable way out of the portability concerns from GNU/Linux becoming more of a closed environment to the point of instigating a massive developer schism? This still raises the question of why it would need to reside in the init daemon, but I suppose crass architectural decisions are not a stranger to Linux developers.
For example: imagine that systemd decides to integrate deeply with GNOME, to the point that you now have an integrated "systemd+glib+GNOME platform" with a unified API
Such a decision might just backfire quite tremendously and prove to be systemd's overturning, though it does sound like a similar integration is very much in the spirit of the project. Then, yes, radical surgery would occur, but it may not be quite as dramatic as branded zones and simply an abandonment of systemd and related Freedesktop-ware.
RedHat and Debian to agree
Debian aren't really that large of a player, more of a passive target platform. Being a non-affiliated foundation, they're generally steered into things by whoever is on the committee and seldom ever lead themselves. So, no agreement strictly necessary.
Ah, correct.
I believe that that friction will eventually cause developers to want to create an init daemon with branded zones to supercede systemd, where emulating systemd-as-it-stands (for apps built to expect it) would be one of that multi-runtime system's goals. With the POSIX people and the bigcorp Linux providers pulling in opposite directions, a multi-runtime Linux would be the only Nash equilibrium. (Though it'd need someone like FreeDesktop to suggest it and start a working group for it, because neither "side" would care about it on their own.)
It's a hypothesis, then. Plausible, certainly. I assume you're presenting this as a probable way out of the portability concerns from GNU/Linux becoming more of a closed environment to the point of instigating a massive developer schism? This still raises the question of why it would need to reside in the init daemon, but I suppose crass architectural decisions are not a stranger to Linux developers.
For example: imagine that systemd decides to integrate deeply with GNOME, to the point that you now have an integrated "systemd+glib+GNOME platform" with a unified API
Such a decision might just backfire quite tremendously and prove to be systemd's overturning, though it does sound like a similar integration is very much in the spirit of the project. Then, yes, radical surgery would occur, but it may not be quite as dramatic as branded zones and simply an abandonment of systemd and related Freedesktop-ware.
RedHat and Debian to agree
Debian aren't really that large of a player, more of a passive target platform. Being a non-affiliated foundation, they're generally steered into things by whoever is on the committee and seldom ever lead themselves. So, no agreement strictly necessary.
Also, given http://skarnet.org/software/sdnotify-wrapper.c it becomes possible to both use s6's superior notification approach in code and to be compatible with how systemd handles notifications.
This is exciting to me and I would like to encourage you to make it way more obvious, given I've been playing with s6 for a while and still only managed to notice it existed because of http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/un... - and while perhaps that's due to my own stupidity, likely there will be other similarly stupid people who'd also value this information ;)
This is exciting to me and I would like to encourage you to make it way more obvious, given I've been playing with s6 for a while and still only managed to notice it existed because of http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/un... - and while perhaps that's due to my own stupidity, likely there will be other similarly stupid people who'd also value this information ;)
Well, note that the article is mostly about the core systemd service management part. The article also clearly studies quality of design and implementation, accepting core goals of systemd core as they are.
One problem with «shinier, newer, better, securer systemd» is that many systemd opponents want something less shiny. And systemd is big and shiny enough that _even_ sysvinit is better.
Most arguments for sysvinit you list are, for example, arguments for s6: s6 is just a supervision suite with no other ambitions, PID 1 during the normal operation (not boot/shutdown) is s6-svscan — a 32 KiB executable, and the part that requires debugging — service launch scripts — can still be a shell script.
I am a systemd hater (at least for my patterns of desktop use — systemd can be mostly fine on some servers) and my opinion of systemd is that it is mediocre as a service management, has bad logging system bundled (is https://bugs.freedesktop.org/show_bug.cgi?id=69934#c7 fixed? can the systemd-bundled logging system give me the last 20 lines of the log without reading all 100 MiB of the log file — it previously could not?), and breaks my workflow hard (nothing similar to startx can be implemented by design http://lists.freedesktop.org/archives/systemd-devel/2012-Feb... for example), and systemd updates make systemd try to control more of the system which I explicitly do not want controlled automatically (I have a very handy binding for standby, and I do not want the notebook to suspend on lid close; I have multiple network configuration scripts for each of three typical locations and choose between them depending on whether I plan to move around more than transfer multiple-GiB files; killing all this autoconfig requires work).
I would probably use uselessd just to avoid bothering if NixOS shipped it instead of full systemd; it doesn't so it is more comfortable for me to use a NixPkgs-based system with init=/bin/sh and minimal hand-written init scripts than stock systemd. I was capable to use systemd almost fine until systemd has ensured that startx-like functionality is impossible to get.
Edit: a typo
One problem with «shinier, newer, better, securer systemd» is that many systemd opponents want something less shiny. And systemd is big and shiny enough that _even_ sysvinit is better.
Most arguments for sysvinit you list are, for example, arguments for s6: s6 is just a supervision suite with no other ambitions, PID 1 during the normal operation (not boot/shutdown) is s6-svscan — a 32 KiB executable, and the part that requires debugging — service launch scripts — can still be a shell script.
I am a systemd hater (at least for my patterns of desktop use — systemd can be mostly fine on some servers) and my opinion of systemd is that it is mediocre as a service management, has bad logging system bundled (is https://bugs.freedesktop.org/show_bug.cgi?id=69934#c7 fixed? can the systemd-bundled logging system give me the last 20 lines of the log without reading all 100 MiB of the log file — it previously could not?), and breaks my workflow hard (nothing similar to startx can be implemented by design http://lists.freedesktop.org/archives/systemd-devel/2012-Feb... for example), and systemd updates make systemd try to control more of the system which I explicitly do not want controlled automatically (I have a very handy binding for standby, and I do not want the notebook to suspend on lid close; I have multiple network configuration scripts for each of three typical locations and choose between them depending on whether I plan to move around more than transfer multiple-GiB files; killing all this autoconfig requires work).
I would probably use uselessd just to avoid bothering if NixOS shipped it instead of full systemd; it doesn't so it is more comfortable for me to use a NixPkgs-based system with init=/bin/sh and minimal hand-written init scripts than stock systemd. I was capable to use systemd almost fine until systemd has ensured that startx-like functionality is impossible to get.
Edit: a typo
I suspect the startx issue is related to why they introduced the su alternative recently, PAM based session tracking.
> I would happily admit there is extensive room for improvement in systemd, but we UNIX folks have been working with worse-is-better and rough-consensus-and-running-code for decades.
You're understanding "worse is better" backwards. It means "starkly simple to the point of being initially off-putting is better". In this context, it means (in part) "be a big boy and learn how to write a shell script". (Not to be too condescending, but I think that captures the Unix perspective here.)
You're understanding "worse is better" backwards. It means "starkly simple to the point of being initially off-putting is better". In this context, it means (in part) "be a big boy and learn how to write a shell script". (Not to be too condescending, but I think that captures the Unix perspective here.)
Yes. I like how you use the word "initially".
Read it through but hard to grasp all the details. One thing I'm sure is that for non-systemd "legacy" init systems I know what I'm doing, for systemd I most of the time don't know what's going on there, it's still like a blind art after using it for a while, overly abstracted might be one reason for that.
I think part of the reason for this may be that some types of systems lend themselves to exploration and gradual learning through use, while others, being very tightly coupled, generally require a larger upfront investment in learning how they function as a whole, since any one subcomponent is only marginally useful without the rest.
This is largely how I remembered studying for Microsoft certification back in the day to be. There was only so much you could easily learn from playing around with the system. At some point, well written technical documentation was, if not required, extremely helpful in making connections between how systems worked together, or explaining exactly what a component was or was not capable of. I don't think this is inherently worse or better, just different, and with its own advantages and disadvantages. For example, people who went through this training seemed to have a very high base-level of understanding of how to configure these systems, what they could or could not do, and how to troubleshoot them, but not necessarily how they functioned in every case. Conversely, the loosely coupled individual components of traditional UNIX service handlers (sysvinit, inetd) promoted people learning how the actually work, and what they do, but with scattered and not entirely comprehensive documentation on how they worked. The average person responsible for using these will have vastly different levels of skill and knowledge about them, but with that comes how it actually functions, because that's needed to troubleshoot problems, since it's so exposed.
The feeling I get mostly when reading about systemd is that it's very much in the vein of MS products. Like I said, that isn't necessarily a bad thing, but it is very different than a lot of UNIX admins are used to.
This is largely how I remembered studying for Microsoft certification back in the day to be. There was only so much you could easily learn from playing around with the system. At some point, well written technical documentation was, if not required, extremely helpful in making connections between how systems worked together, or explaining exactly what a component was or was not capable of. I don't think this is inherently worse or better, just different, and with its own advantages and disadvantages. For example, people who went through this training seemed to have a very high base-level of understanding of how to configure these systems, what they could or could not do, and how to troubleshoot them, but not necessarily how they functioned in every case. Conversely, the loosely coupled individual components of traditional UNIX service handlers (sysvinit, inetd) promoted people learning how the actually work, and what they do, but with scattered and not entirely comprehensive documentation on how they worked. The average person responsible for using these will have vastly different levels of skill and knowledge about them, but with that comes how it actually functions, because that's needed to troubleshoot problems, since it's so exposed.
The feeling I get mostly when reading about systemd is that it's very much in the vein of MS products. Like I said, that isn't necessarily a bad thing, but it is very different than a lot of UNIX admins are used to.
>Conversely, the loosely coupled individual components of traditional UNIX service handlers (sysvinit, inetd) promoted people learning how the actually work, and what they do, but with scattered and not entirely comprehensive documentation on how they worked.
Can you tease out this distinction a bit more? It reads like you are saying, due to their nature, people learned how to use Unix service handlers, without a good understanding of how they were designed. It could also mean that people learned how Unix service handlers worked by using them, through trial and error, because the documentation was scattered and not entirely comprehensive.
Can you tease out this distinction a bit more? It reads like you are saying, due to their nature, people learned how to use Unix service handlers, without a good understanding of how they were designed. It could also mean that people learned how Unix service handlers worked by using them, through trial and error, because the documentation was scattered and not entirely comprehensive.
The second interpretation. The documentation is not entirely comprehensive because interface between components is not only a grey area of accountability, but also continuously changing. When the system is composed of multiple distinct sub-programs with their own development teams, update cycles and planned feature sets, what is possible at any one point in time is largely a matter of which versions of those sub-programs you are using, as well as what the root program (e.g.sysvinit) allows in it's version. There is no one set of features, nor one set of bugs, nor one set of directions. The best you can get in this regard is whatever your distro put together for the combination they chose, or alternatively to read the docs for every sub-program and intuit what new capabilities may exist in how they interact.
I think the clearest example of this would be everything that makes up the core Linux environment, compared OpenBSD. OpenBSD has amazing documentation, and the systems are all designed to work with eachother because they were all designed and developed as a working whole. Linux, on the other hand, often exposes interfaces through the kernel, and some other set of developers (or multiple set of developers) create systems to interface with it. As such, the documentation can be haphazard or near non-existent. That's not to say the Linux methodology doesn't have it's advantages. For example, I think there's a better feedback loop in Linux in determining what is needed from the kernel, and what can be done from it in userspace. It's a more market-like system, and sees rapid advancement.
I think the clearest example of this would be everything that makes up the core Linux environment, compared OpenBSD. OpenBSD has amazing documentation, and the systems are all designed to work with eachother because they were all designed and developed as a working whole. Linux, on the other hand, often exposes interfaces through the kernel, and some other set of developers (or multiple set of developers) create systems to interface with it. As such, the documentation can be haphazard or near non-existent. That's not to say the Linux methodology doesn't have it's advantages. For example, I think there's a better feedback loop in Linux in determining what is needed from the kernel, and what can be done from it in userspace. It's a more market-like system, and sees rapid advancement.
Thank you!
Very interesting article, refreshing as well to see a criticism of systemd based on technical details. Unlike most other criticisms of systemd, this author does not resort to personal attacks and insults against Lennart Poettering or fear-mongering appeals to tradition. I still think systemd is an improvement over the previous sysvinit, but the author seems to have found some real conceptual flaws with systemd that are somewhat unsettling.
I hope this article can start some actually useful conversations about init systems that do not fall into the pitfalls of FUD and negativity that seem to have surrounded this topic in the past.
I hope this article can start some actually useful conversations about init systems that do not fall into the pitfalls of FUD and negativity that seem to have surrounded this topic in the past.
My explanation for the lack of technical (rather than political) articles examining systemd is that it's very difficult to wrap your mind around because it carries so many ideas. A quote from an article linked within the linked article:
> All of the above definitions are technically correct. None of them are complete.
> It is thus that debates are doomed to go nowhere, as no one can agree on a single definition of systemd, and the developers have been largely unhelpful in that matter. Vagueness can be a strength, in that it allows for people to divide and conquer, but it is also unproductive and leads to unnecessary bloodshed.
Slowly, surely, people will begin to understand what systemd is. Only then will people be able to figure out whether they want it or not.
> All of the above definitions are technically correct. None of them are complete.
> It is thus that debates are doomed to go nowhere, as no one can agree on a single definition of systemd, and the developers have been largely unhelpful in that matter. Vagueness can be a strength, in that it allows for people to divide and conquer, but it is also unproductive and leads to unnecessary bloodshed.
Slowly, surely, people will begin to understand what systemd is. Only then will people be able to figure out whether they want it or not.
The major problem right now if one decides to use a systemd alternative is that most packages are shipping with systemd unit files, but will obviously lack support for other init implementations.
I like dmd, implemented in Scheme, which is part of the GuixSD distribution.
I like dmd, implemented in Scheme, which is part of the GuixSD distribution.
1. The solution to this is to write a translator from systemd units (those of .service type) to the desired kind of unit. I haven't looked into dmd, but this should be possible at least for the majority of units.
2. This was going to be a problem in any case i.e. had most distros switched to openRC one would still have been faced with the task of producing a suitable collection of service configuration files for dmd and then adapting them to dmd's specfic featureset.
2. This was going to be a problem in any case i.e. had most distros switched to openRC one would still have been faced with the task of producing a suitable collection of service configuration files for dmd and then adapting them to dmd's specfic featureset.
Yeah, well, if it's to go from unit files to Guile code, that's not what I'd consider an improvement. Having configuration files for use cases where the requirements are simple enough is nice.
Of course, when they're not, you end up with an abomination like MSBuild, but that's not the case here.
Of course, when they're not, you end up with an abomination like MSBuild, but that's not the case here.
True, but if that is the case you won't be running dmd in the first place. And automated conversion/runtime translation will be a possibility even with different service managers, e.g. systemXVI.
I'm not really convinced by the second, in that I believe that nothing in the article proves that service management is something too complex to be done via configuration files.
I'm not really convinced by the second, in that I believe that nothing in the article proves that service management is something too complex to be done via configuration files.
The article that lies herein, was, in a manner of speaking (but not literally speaking - 'in a manner of speaking' is just a phrase), so overly verbose in many places that it drowns the reader in verbal diarrhea as thick as pea soup. That is to say, I, as a reader, felt that the author (of this article) could have made these points in a more reachable, succinct fashion.
I would, as a general principle, be cautious (in the sense of not being hasty) to dismiss in a manner reminiscent of Parkinson's law of triviality (insofar as the latter pertains to the bikeshed principle) the literary style of the article in question, since the common interpretation of excessive verbosity is flawed, in this case being seldom more than a preference with little in the way of any concrete justification that would be of direct relevance to the topic at hand, the topic being an architectural treatise. Any reservations about succinctness, or a perceived lack thereof, must therefore be qualified and left over for further analysis to draw proper and conceptually robust remarks given the evidence presented.
(Yes, the writing style is somewhat obtuse. You'll have to excuse that. The topic itself is obscure, so I suppose it fits.)
(Yes, the writing style is somewhat obtuse. You'll have to excuse that. The topic itself is obscure, so I suppose it fits.)
My response from another discussion on this article:
Much of these deficiencies boil down to the design criteria for systemd, which is basically to avoid forcing the system administrator to perform dependency graph resolution. This was a direct response to Upstart, which pushed this work to the administrator (or distro packager) by design.
Thus the declarative syntax for unit files, which do not explicitly control dependency resolution, parallelization, or startup order, which is simply not necessary or cared about for 99% of use cases. The 1% case there is for situations where you should not use a prebaked init system in the first place, such as when you absolutely need deterministic boots, i.e. in an embedded or realtime context.
> In systemd, however, the execution state is not based on explicit chain loading, but on serializing unit file options to a private ExecContext structure overlayed into Unit objects and set for service unit types.
This is not a problem with systemd's design. This is a problem with software written without the assumption of reliable process supervision, which is a reasonable assumption given the history of UNIX daemonization. systemd takes the (apparently controversial) stance that processes should not manage their own execution environment, or that of their dependencies. It provides numerous prebuilt hacks to make things work for broken software, but nobody should expect systemd to design for 100% of situations where processes do Weird Shit™.
And yes, I'm including "delegated service restarters" under this umbrella. The need for something like this absolutely smells of poor design. It also flies in the face of the author's preference for deterministic execution behavior, because by definition your process will execute differently depending on if it was started or restarted.
The link to "systemd house of horrors" [1] only proves this point.
> A supervisor insisting on programmatic accommodations from service writers is not the most desirable state of affairs. A rarely discussed alternative to the two common approaches is, again as with startup speed optimization minus parallelism, checkpoint/restore: checkpoint a process image from a point where initialization is known to be complete and overlay it on startup, using a tool such as DMTCP or CRIU.
I don't know about anyone else, but this sounds like an absolute nightmare. For one, it relies on the service exposing enough information to be checkpointed at the right time; at which point, you might as well call sd_notify or hook in a script to do it for you, so this is not a reduction in requiring programmatic accommodations from service writers. Two, process-snapshotting is not a panacea, and is a nightmare to debug when it breaks your expectations; you'll need to test all sorts of process state, including open pipes/fds/sockets, signals, system calls (!), shmem buffers, etc. At which point it's best to just let the service notify you directly when the process is done. Which is a facility that systemd provides.
> Of course we also have the issue of circular dependencies in the systemd architecture itself. We have the init, process manager, process supervisor, cgroup writer, local service tools, the Unit object structure (which might benefit from being made a protocol), timers, mounts, automounts and swaps all in the same module with ill-defined boundaries.
A worthy criticism. I would also like to see a bit more modularization from systemd, but I can also see the point of systemd developers not expending effort to do so. For one, it would drastically increase the number of contexts for testing (i.e. can we verify the service manager works without the cgroups module loaded, etc). I think systemd is at the point where it's rock-solid for all of its use cases, and a pluggable/composable architecture would have delayed this situation for not very much benefit.
In short, systemd imposes its opinions about process init and supervision. Whether or not this is reasonable is up for debate. However, it is very useful to have both a strong opinion and a solid implementation of this opinion.
IMO, the idealized world that systemd supports (i.e. non-forking services, initialization notification, strict execution environment and dependency declaration, centralized and opaque init/dependency-resolution/parallelization) is a vast improvement over the status quo and actually improves the lives of service writers (by removing an entire class of responsibilities), so I think it's a reasonable opinion and thus I think systemd has a reasonable design and architecture.
There are many known technical deficiencies with the implementation that are mostly being worked on, but I'm not seeing anything damning of the whole idea.
[1] http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/sy...
Much of these deficiencies boil down to the design criteria for systemd, which is basically to avoid forcing the system administrator to perform dependency graph resolution. This was a direct response to Upstart, which pushed this work to the administrator (or distro packager) by design.
Thus the declarative syntax for unit files, which do not explicitly control dependency resolution, parallelization, or startup order, which is simply not necessary or cared about for 99% of use cases. The 1% case there is for situations where you should not use a prebaked init system in the first place, such as when you absolutely need deterministic boots, i.e. in an embedded or realtime context.
> In systemd, however, the execution state is not based on explicit chain loading, but on serializing unit file options to a private ExecContext structure overlayed into Unit objects and set for service unit types.
This is not a problem with systemd's design. This is a problem with software written without the assumption of reliable process supervision, which is a reasonable assumption given the history of UNIX daemonization. systemd takes the (apparently controversial) stance that processes should not manage their own execution environment, or that of their dependencies. It provides numerous prebuilt hacks to make things work for broken software, but nobody should expect systemd to design for 100% of situations where processes do Weird Shit™.
And yes, I'm including "delegated service restarters" under this umbrella. The need for something like this absolutely smells of poor design. It also flies in the face of the author's preference for deterministic execution behavior, because by definition your process will execute differently depending on if it was started or restarted.
The link to "systemd house of horrors" [1] only proves this point.
> A supervisor insisting on programmatic accommodations from service writers is not the most desirable state of affairs. A rarely discussed alternative to the two common approaches is, again as with startup speed optimization minus parallelism, checkpoint/restore: checkpoint a process image from a point where initialization is known to be complete and overlay it on startup, using a tool such as DMTCP or CRIU.
I don't know about anyone else, but this sounds like an absolute nightmare. For one, it relies on the service exposing enough information to be checkpointed at the right time; at which point, you might as well call sd_notify or hook in a script to do it for you, so this is not a reduction in requiring programmatic accommodations from service writers. Two, process-snapshotting is not a panacea, and is a nightmare to debug when it breaks your expectations; you'll need to test all sorts of process state, including open pipes/fds/sockets, signals, system calls (!), shmem buffers, etc. At which point it's best to just let the service notify you directly when the process is done. Which is a facility that systemd provides.
> Of course we also have the issue of circular dependencies in the systemd architecture itself. We have the init, process manager, process supervisor, cgroup writer, local service tools, the Unit object structure (which might benefit from being made a protocol), timers, mounts, automounts and swaps all in the same module with ill-defined boundaries.
A worthy criticism. I would also like to see a bit more modularization from systemd, but I can also see the point of systemd developers not expending effort to do so. For one, it would drastically increase the number of contexts for testing (i.e. can we verify the service manager works without the cgroups module loaded, etc). I think systemd is at the point where it's rock-solid for all of its use cases, and a pluggable/composable architecture would have delayed this situation for not very much benefit.
In short, systemd imposes its opinions about process init and supervision. Whether or not this is reasonable is up for debate. However, it is very useful to have both a strong opinion and a solid implementation of this opinion.
IMO, the idealized world that systemd supports (i.e. non-forking services, initialization notification, strict execution environment and dependency declaration, centralized and opaque init/dependency-resolution/parallelization) is a vast improvement over the status quo and actually improves the lives of service writers (by removing an entire class of responsibilities), so I think it's a reasonable opinion and thus I think systemd has a reasonable design and architecture.
There are many known technical deficiencies with the implementation that are mostly being worked on, but I'm not seeing anything damning of the whole idea.
[1] http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/sy...
It appears you received a response here: https://www.reddit.com/r/linux/comments/3od733/structural_an...
You do indeed misunderstand the point about execution composition. Chain loading is indeed configuration-related, not service-related. It's a much more flexible and visible method than internal structures. Delegated restarters then can be deterministic as they reuse process management.
C/R solutions don't require special integration, that's the point. They work on unmodified processes, unlike readiness notification. This is I think a reasonable trade-off for the potential decrease in reliability, as the feature comes with a double advantage of startup optimization and then readiness notification is generally not that critical, in light of regular Unix process semantics. C/R is also constantly improving and reasonably stable for base use cases these days.
You do indeed misunderstand the point about execution composition. Chain loading is indeed configuration-related, not service-related. It's a much more flexible and visible method than internal structures. Delegated restarters then can be deterministic as they reuse process management.
C/R solutions don't require special integration, that's the point. They work on unmodified processes, unlike readiness notification. This is I think a reasonable trade-off for the potential decrease in reliability, as the feature comes with a double advantage of startup optimization and then readiness notification is generally not that critical, in light of regular Unix process semantics. C/R is also constantly improving and reasonably stable for base use cases these days.
You seem to dislike the idea of asking for notification support in the daemons themselves. Granted, it's impractical, but it's the only way to avoid race conditions which are rare for most services, but very real for a few programs that may take ages to get ready.
How would you address the problem of slow starters without support from the daemons, while keeping a "the dependency graph is always respected" invariant?
Right, if the problem is slow starters (say, RabbitMQ), then restoring from an image where initialization has already been complete would resolve it. For more complicated cases where there is a race with acquiring some resource, then a readiness notification mechanism could be supported as a niche feature. I'm still not convinced of it in the general case.
But how would you go with restoring such an image? It won't be instant, there will still be a window where the service manager is continuing its work but the image hasn't been restored yet. One way or the other, you need a synchronization mechanism, so how would you design such a mechanism without support from the daemon?
You don't need a complicated case to have a race with acquiring some resource. A server opening a socket is providing a resource; clients connecting to that socket are using it. Clients should not be started before the server is listening to the socket. How would you solve this case - are you advocating socket activation by the service manager? ;)
You don't need a complicated case to have a race with acquiring some resource. A server opening a socket is providing a resource; clients connecting to that socket are using it. Clients should not be started before the server is listening to the socket. How would you solve this case - are you advocating socket activation by the service manager? ;)
Thank you for doing this - I'm going to read this with a lot of interest :-)
Do return with commentary.
tl;dr: the author argues that:
* systemd is more similar to a traditional job scheduler than a traditional init system.
* systemd should not parse text files in pid1 because it might be a security risk.
* Because systemd handles order dependencies, it is sometimes susceptible to "ordering cycles" where a clear ordering cannot be established, "dependency loops" where jobs are continuously dropped and requeued, and race conditions if dependencies are accidentally underspecified.
* "Imbalance between promoting laziness or eagerness": launchd, a predecessor of systemd, operated purely "lazily," launching services only when other services needed them. systemd also supports launching services eagerly, which the author argues is more complex.
* Checkpointing a process image could have been used instead of systemd's readiness notification mechanism (?)
* Systemd does not have a plugin system
* Journald is criticized because it does too much
I would counter that:
* I agree that systemd is similar to a job scheduler, but I don't necessarily believe that that is a bad thing. Manual scheduling of jobs for startup and shutdown like we did in the rc.sh days is complex and error-prone.
* The text files systemd is parsing are owned by root, so even if the parser is exploitable, only root can take advantage? Seems like a weak criticism.
* I agree that dependency ordering makes things more complex. But upstart and launchd have the same issues. It seems to be a tradeoff worth making.
* I don't understand the argument against supporting both laziness and eagerness in dependencies. It seems like some things are naturally modelled eagerly, like needing to perform a bunch of somewhat unrelated actions to suspend the system. And some things are better handled lazily, like setting up a FUSE filesystem when a USB stick is inserted that needs it. Shouldn't we use the model that makes the most sense?
* I don't think checkpointing a process image can ever really replace having a notification system. Even if the checkpointing code could be make bulletproof somehow, some processes deal with state in the external world, or with hardware, that makes checkpointing infeasible.
* Regarding a plugin system: Systemd has the ability to run shell scripts, which can be useful in filling in gaps in functionality. I don't think a more complicated plugin system would be a good idea since it would add a lot of complexity (and potentially instability.)
* I didn't understand the criticism of journald, maybe someone can elaborate. The author presented some alternate approaches but I missed why these were better (other than the handwavey argument that journald was "a bottleneck")
I enjoyed reading this, and it's nice to see some more reasoned criticism of systemd. I feel like the prose got a little bit purple at times. We had to spend 10 paragraphs "descend[ing] into an inferno with the same dead horse talking points", "culminating into some rather heterodox conclusions", and "progressively introducing and elucidating on concepts, applying some a priori reasoning at times, and backtracking to derive conclusions or reiterate on prior stated knowledge" before we even saw a word of argument! This needed an editor...
* systemd is more similar to a traditional job scheduler than a traditional init system.
* systemd should not parse text files in pid1 because it might be a security risk.
* Because systemd handles order dependencies, it is sometimes susceptible to "ordering cycles" where a clear ordering cannot be established, "dependency loops" where jobs are continuously dropped and requeued, and race conditions if dependencies are accidentally underspecified.
* "Imbalance between promoting laziness or eagerness": launchd, a predecessor of systemd, operated purely "lazily," launching services only when other services needed them. systemd also supports launching services eagerly, which the author argues is more complex.
* Checkpointing a process image could have been used instead of systemd's readiness notification mechanism (?)
* Systemd does not have a plugin system
* Journald is criticized because it does too much
I would counter that:
* I agree that systemd is similar to a job scheduler, but I don't necessarily believe that that is a bad thing. Manual scheduling of jobs for startup and shutdown like we did in the rc.sh days is complex and error-prone.
* The text files systemd is parsing are owned by root, so even if the parser is exploitable, only root can take advantage? Seems like a weak criticism.
* I agree that dependency ordering makes things more complex. But upstart and launchd have the same issues. It seems to be a tradeoff worth making.
* I don't understand the argument against supporting both laziness and eagerness in dependencies. It seems like some things are naturally modelled eagerly, like needing to perform a bunch of somewhat unrelated actions to suspend the system. And some things are better handled lazily, like setting up a FUSE filesystem when a USB stick is inserted that needs it. Shouldn't we use the model that makes the most sense?
* I don't think checkpointing a process image can ever really replace having a notification system. Even if the checkpointing code could be make bulletproof somehow, some processes deal with state in the external world, or with hardware, that makes checkpointing infeasible.
* Regarding a plugin system: Systemd has the ability to run shell scripts, which can be useful in filling in gaps in functionality. I don't think a more complicated plugin system would be a good idea since it would add a lot of complexity (and potentially instability.)
* I didn't understand the criticism of journald, maybe someone can elaborate. The author presented some alternate approaches but I missed why these were better (other than the handwavey argument that journald was "a bottleneck")
I enjoyed reading this, and it's nice to see some more reasoned criticism of systemd. I feel like the prose got a little bit purple at times. We had to spend 10 paragraphs "descend[ing] into an inferno with the same dead horse talking points", "culminating into some rather heterodox conclusions", and "progressively introducing and elucidating on concepts, applying some a priori reasoning at times, and backtracking to derive conclusions or reiterate on prior stated knowledge" before we even saw a word of argument! This needed an editor...
Actually the job scheduler is one aspect, complimentary to the object-oriented resource management which I made quite clear was the main point.
Regarding file ownership, not necessarily. If you run it as a session-based init, they'd be owned by the user it serves as an agent to. (see dirs: http://www.freedesktop.org/software/systemd/man/systemd.unit...)
Upstart and launchd do not have the same issues. Upstart doesn't have dependencies, but named preconditions and postconditions. launchd has neither that nor dependencies. They have many problems of their own, but different ones.
My point regarding laziness v eagerness is the way systemd implements multi-paradigm introduces undesirable externalities from both approaches, as I said: "In systemd, one must endure both the non-determinism from aggressive parallelism via dependency information, the object network needing the transaction and job metaphors, all while having its lazy features be thoroughly restricted to its internal Manager and not a flexible toolkit such as UCSPI."
I'd assume the limited use of C/R to simply cut down initialization times is feasible: http://criu.org/Usage_scenarios#Slow-boot_services_speed_up
Running shell scripts doesn't mean much when the underlying system has a large surface and ill-defined module boundaries. A plugin system most certainly does not have to be complicated: look at finit and initng for examples.
What about journald did you not understand? You lose control over the separate stages of collection, rotation and processing. Particularly having custom postprocessing is important for interoperability with foreign services that might expect certain format constraints. Having the logs be per-service instead of globally snarfed is also saner.
Yes, I did go overboard on the reservations. It's a flame war topic, so I felt it necessary as a precautionary measure.
Regarding file ownership, not necessarily. If you run it as a session-based init, they'd be owned by the user it serves as an agent to. (see dirs: http://www.freedesktop.org/software/systemd/man/systemd.unit...)
Upstart and launchd do not have the same issues. Upstart doesn't have dependencies, but named preconditions and postconditions. launchd has neither that nor dependencies. They have many problems of their own, but different ones.
My point regarding laziness v eagerness is the way systemd implements multi-paradigm introduces undesirable externalities from both approaches, as I said: "In systemd, one must endure both the non-determinism from aggressive parallelism via dependency information, the object network needing the transaction and job metaphors, all while having its lazy features be thoroughly restricted to its internal Manager and not a flexible toolkit such as UCSPI."
I'd assume the limited use of C/R to simply cut down initialization times is feasible: http://criu.org/Usage_scenarios#Slow-boot_services_speed_up
Running shell scripts doesn't mean much when the underlying system has a large surface and ill-defined module boundaries. A plugin system most certainly does not have to be complicated: look at finit and initng for examples.
What about journald did you not understand? You lose control over the separate stages of collection, rotation and processing. Particularly having custom postprocessing is important for interoperability with foreign services that might expect certain format constraints. Having the logs be per-service instead of globally snarfed is also saner.
Yes, I did go overboard on the reservations. It's a flame war topic, so I felt it necessary as a precautionary measure.
Regarding file ownership, not necessarily. If you run it as a session-based init, they'd be owned by the user it serves as an agent to.
Interesting. I didn't realize systemd had a "user mode" where it doesn't run as pid 1. Does systemd run as root (or root equivalent) in this mode? I wasn't able to find this from a quick skim of the information available online.
Upstart and launchd do not have the same issues. Upstart doesn't have dependencies, but named preconditions and postconditions. launchd has neither that nor dependencies. They have many problems of their own, but different ones.
Sorry, but I have to disagree. The terminology may be different, but ultimately upstart's "start on" directive establishes a dependency. Once you have a dependency you can have circular dependencies. And once you rely on the init systems to establish ordering rather than having everything just start up in a deterministic order in simple single-threaded script, you can have race conditions where things happen to work on one machine but not another. I'm not as familiar with launchd; maybe its socket activation makes the circular ordering harder to hit, but fundamentally there can always be races.
I'd assume the limited use of C/R to simply cut down initialization times is feasible: http://criu.org/Usage_scenarios#Slow-boot_services_speed_up
Perhaps for some processes this is feasible, but nothing on that site addresses my objections above: "some processes deal with state in the external world, or with hardware, that makes checkpointing infeasible." You will always need a notification mechanism even if you have checkpoint and restore. And why have the complexity of checkpointing if you have a working notification system?
Running shell scripts doesn't mean much when the underlying system has a large surface and ill-defined module boundaries. A plugin system most certainly does not have to be complicated: look at finit and initng for examples.
From a user's point of view, I do not want a plugin system. I want systemd to work the same on SUSE vs. debian vs. Red Hat. It is usually better to put things into the base system than to have them live in a half-supported state outside the tree. That's why web browsers are killing plugins: in order to be useful, they have to reach deep into the guts of the program, which is not a place I want 3rd party code.
What about journald did you not understand? You lose control over the separate stages of collection, rotation and processing. Particularly having custom postprocessing is important for interoperability with foreign services that might expect certain format constraints.
I don't understand why I would want separate daemons for collection and rotation, at least for dmsg, kmsg, and syslog. logrotate always had a bunch of race conditions; it was never a great design.
I agree that journald doesn't do custom processing. However, I'm a little unsure whether I would really want it to, or whether I'd be better off running my own syslogd implementation at that point. That is still an option, even when you are using the rest of systemd.
Having the logs be per-service instead of globally snarfed is also saner.
You can always set up important services to log to a file rather than to syslog.
My real criticism of systemd is that they made no effort at portability outside of Linux (and I don't buy their arguments about why it's too hard.) It would have been a lot less controversial if they had at least pointed out how the BSDs could add support (even if they themselves didn't do the work).
Interesting. I didn't realize systemd had a "user mode" where it doesn't run as pid 1. Does systemd run as root (or root equivalent) in this mode? I wasn't able to find this from a quick skim of the information available online.
Upstart and launchd do not have the same issues. Upstart doesn't have dependencies, but named preconditions and postconditions. launchd has neither that nor dependencies. They have many problems of their own, but different ones.
Sorry, but I have to disagree. The terminology may be different, but ultimately upstart's "start on" directive establishes a dependency. Once you have a dependency you can have circular dependencies. And once you rely on the init systems to establish ordering rather than having everything just start up in a deterministic order in simple single-threaded script, you can have race conditions where things happen to work on one machine but not another. I'm not as familiar with launchd; maybe its socket activation makes the circular ordering harder to hit, but fundamentally there can always be races.
I'd assume the limited use of C/R to simply cut down initialization times is feasible: http://criu.org/Usage_scenarios#Slow-boot_services_speed_up
Perhaps for some processes this is feasible, but nothing on that site addresses my objections above: "some processes deal with state in the external world, or with hardware, that makes checkpointing infeasible." You will always need a notification mechanism even if you have checkpoint and restore. And why have the complexity of checkpointing if you have a working notification system?
Running shell scripts doesn't mean much when the underlying system has a large surface and ill-defined module boundaries. A plugin system most certainly does not have to be complicated: look at finit and initng for examples.
From a user's point of view, I do not want a plugin system. I want systemd to work the same on SUSE vs. debian vs. Red Hat. It is usually better to put things into the base system than to have them live in a half-supported state outside the tree. That's why web browsers are killing plugins: in order to be useful, they have to reach deep into the guts of the program, which is not a place I want 3rd party code.
What about journald did you not understand? You lose control over the separate stages of collection, rotation and processing. Particularly having custom postprocessing is important for interoperability with foreign services that might expect certain format constraints.
I don't understand why I would want separate daemons for collection and rotation, at least for dmsg, kmsg, and syslog. logrotate always had a bunch of race conditions; it was never a great design.
I agree that journald doesn't do custom processing. However, I'm a little unsure whether I would really want it to, or whether I'd be better off running my own syslogd implementation at that point. That is still an option, even when you are using the rest of systemd.
Having the logs be per-service instead of globally snarfed is also saner.
You can always set up important services to log to a file rather than to syslog.
My real criticism of systemd is that they made no effort at portability outside of Linux (and I don't buy their arguments about why it's too hard.) It would have been a lot less controversial if they had at least pointed out how the BSDs could add support (even if they themselves didn't do the work).
Once you have a dependency you can have circular dependencies.
Then the question becomes whether an architecture obscures or exacerbates them, and also how it deals with them. The systemd object model does not fare well, as discussed.
(Also compare a system like s6-rc or serel where dependency graphs are compiled from manifests and consistency checked before being applied. The graph is persistent configuration and not an ephemeral artifact.)
And once you rely on the init systems to establish ordering rather than having everything just start up in a deterministic order in simple single-threaded script, you can have race conditions where things happen to work on one machine but not another.
Systems having ordering dependencies does not inherently affect their determinism, the startup discipline (sequential v. parallel) primarily does, and then there are different ways to tackle parallelism. systemd's execution model and transactional object system exacerbate non-determinism in ordering results.
I don't understand why I would want separate daemons for collection and rotation, at least for dmsg, kmsg, and syslog. logrotate always had a bunch of race conditions; it was never a great design.
They're not separate daemons.
My real criticism of systemd is that they made no effort at portability outside of Linux (and I don't buy their arguments about why it's too hard.)
That's a horrible criticism of systemd. Any cursory examination of systemd will quickly reveal the task is next to insurmountable without doing what amounts to a full reimplementation.
Then the question becomes whether an architecture obscures or exacerbates them, and also how it deals with them. The systemd object model does not fare well, as discussed.
(Also compare a system like s6-rc or serel where dependency graphs are compiled from manifests and consistency checked before being applied. The graph is persistent configuration and not an ephemeral artifact.)
And once you rely on the init systems to establish ordering rather than having everything just start up in a deterministic order in simple single-threaded script, you can have race conditions where things happen to work on one machine but not another.
Systems having ordering dependencies does not inherently affect their determinism, the startup discipline (sequential v. parallel) primarily does, and then there are different ways to tackle parallelism. systemd's execution model and transactional object system exacerbate non-determinism in ordering results.
I don't understand why I would want separate daemons for collection and rotation, at least for dmsg, kmsg, and syslog. logrotate always had a bunch of race conditions; it was never a great design.
They're not separate daemons.
My real criticism of systemd is that they made no effort at portability outside of Linux (and I don't buy their arguments about why it's too hard.)
That's a horrible criticism of systemd. Any cursory examination of systemd will quickly reveal the task is next to insurmountable without doing what amounts to a full reimplementation.
> (Also compare a system like s6-rc or serel where dependency graphs are compiled from manifests and consistency checked before being applied. The graph is persistent configuration and not an ephemeral artifact.)
I agree with that point. This would ensure it can start with the last known-good configuration (though there is a chicken-and-egg problem here). As a user of XMonad, it's nice to know that I can mess up my "configuration file" (which is just Haskell code) without worries (unlike with what a WM like Awesome), which is probably the only thing about having Turing-complete configuration language I like.
I agree with that point. This would ensure it can start with the last known-good configuration (though there is a chicken-and-egg problem here). As a user of XMonad, it's nice to know that I can mess up my "configuration file" (which is just Haskell code) without worries (unlike with what a WM like Awesome), which is probably the only thing about having Turing-complete configuration language I like.
Then the question becomes whether an architecture obscures or exacerbates them, and also how it deals with them. The systemd object model does not fare well, as discussed.
What specific aspects of the systemd dependency model do you feel exacerbate race conditions and circular dependencies?
(Also compare a system like s6-rc or serel where dependency graphs are compiled from manifests and consistency checked before being applied. The graph is persistent configuration and not an ephemeral artifact.)
I have seen people discussing systemd error messages about circular dependencies online. It seems that systemd does perform some checking, although I haven't found documentation on what checks it does (I probably missed it). I'm not sure how the dependency graph could be persistent configuration because hardware can be hotplugged these days. For example, are you going to create nodes in the graph for all the hard drives that could possibly exist over the lifetime of the machine?
Systems having ordering dependencies does not inherently affect their determinism, the startup discipline (sequential v. parallel) primarily does, and then there are different ways to tackle parallelism. systemd's execution model and transactional object system exacerbate non-determinism in ordering results.
If the sequential vs. parallel startup discipline "primarily affects determinism" won't all these systems have the same issues with nondeterminism? I don't think anyone is seriously proposing starting up services on boot sequentially in 2015. Or even when a USB stick is inserted. Systems have 12 cores now and people can be very sensitive to boot times.
[lack of portability is] a horrible criticism of systemd. Any cursory examination of systemd will quickly reveal the task is next to insurmountable without doing what amounts to a full reimplementation.
I understand that systemd relies heavily on cgroups and other Linux-specific features for process isolation. And the BSDs are also unlikely to accept GPLed code. However, I feel like they could have at least discussed some way of reducing fragmentation here. As it is, I would be amazed if at least one or two of the BSDs didn't implement a similar init system to systemd in the next few years. And as it is, they probably won't share any code or concepts, unfortunately.
What specific aspects of the systemd dependency model do you feel exacerbate race conditions and circular dependencies?
(Also compare a system like s6-rc or serel where dependency graphs are compiled from manifests and consistency checked before being applied. The graph is persistent configuration and not an ephemeral artifact.)
I have seen people discussing systemd error messages about circular dependencies online. It seems that systemd does perform some checking, although I haven't found documentation on what checks it does (I probably missed it). I'm not sure how the dependency graph could be persistent configuration because hardware can be hotplugged these days. For example, are you going to create nodes in the graph for all the hard drives that could possibly exist over the lifetime of the machine?
Systems having ordering dependencies does not inherently affect their determinism, the startup discipline (sequential v. parallel) primarily does, and then there are different ways to tackle parallelism. systemd's execution model and transactional object system exacerbate non-determinism in ordering results.
If the sequential vs. parallel startup discipline "primarily affects determinism" won't all these systems have the same issues with nondeterminism? I don't think anyone is seriously proposing starting up services on boot sequentially in 2015. Or even when a USB stick is inserted. Systems have 12 cores now and people can be very sensitive to boot times.
[lack of portability is] a horrible criticism of systemd. Any cursory examination of systemd will quickly reveal the task is next to insurmountable without doing what amounts to a full reimplementation.
I understand that systemd relies heavily on cgroups and other Linux-specific features for process isolation. And the BSDs are also unlikely to accept GPLed code. However, I feel like they could have at least discussed some way of reducing fragmentation here. As it is, I would be amazed if at least one or two of the BSDs didn't implement a similar init system to systemd in the next few years. And as it is, they probably won't share any code or concepts, unfortunately.
What specific aspects of the systemd dependency model do you feel exacerbate race conditions and circular dependencies?
Well, the article discusses all of that...
(Speaking of which, JdeBP helpfully linked to Upstart's pre/postconditions not being a proper dependency model, per the developers' testimony, and as I asserted.)
I'm not sure how the dependency graph could be persistent configuration because hardware can be hotplugged these days.
The idea of device units is not well borne out, as I mentioned. You don't need to create an object type out of a udev handle to order after devices or trigger hotplug events (the device node manager itself does fine, of course in the case of systemd, udev is part of the project, so...)
If the sequential vs. parallel startup discipline "primarily affects determinism" won't all these systems have the same issues with nondeterminism?
Not necessarily. It could either be more manageable if there wasn't so much indirection, see above (I'm afraid you just didn't get the article, and your original summary certainly implied as much). Additionally, one could forfeit being aggressively parallel for a more conservative approach like in OpenRC, s6-rc or even Upstart.
* I don't think anyone is seriously proposing starting up services on boot sequentially in 2015. Or even when a USB stick is inserted. Systems have 12 cores now and people can be very sensitive to boot times.*
Parallelism is but a small part of the complicated topic of boot time optimization. And yes, sequential strategies can still be really fast. Much of the speed increase in modern systems is not from parallelism, but forfeiting the overhead of the shell interpreter in favor of direct execution. Checkpointing can do far greater wonders for your boot times than parallelism can, anyhow.
I understand that systemd relies heavily on cgroups and other Linux-specific features for process isolation.
That's only scratching the surface of Linux-specific features in systemd.
As it is, I would be amazed if at least one or two of the BSDs didn't implement a similar init system to systemd in the next few years. And as it is, they probably won't share any code or concepts, unfortunately.
What will come of the BSDs is uncertain. launchd will be used in iXsystems products like PC-BSD and FreeNAS, and there's not much "code sharing" you could do there - it's already a complete product being ported.
Well, the article discusses all of that...
(Speaking of which, JdeBP helpfully linked to Upstart's pre/postconditions not being a proper dependency model, per the developers' testimony, and as I asserted.)
I'm not sure how the dependency graph could be persistent configuration because hardware can be hotplugged these days.
The idea of device units is not well borne out, as I mentioned. You don't need to create an object type out of a udev handle to order after devices or trigger hotplug events (the device node manager itself does fine, of course in the case of systemd, udev is part of the project, so...)
If the sequential vs. parallel startup discipline "primarily affects determinism" won't all these systems have the same issues with nondeterminism?
Not necessarily. It could either be more manageable if there wasn't so much indirection, see above (I'm afraid you just didn't get the article, and your original summary certainly implied as much). Additionally, one could forfeit being aggressively parallel for a more conservative approach like in OpenRC, s6-rc or even Upstart.
* I don't think anyone is seriously proposing starting up services on boot sequentially in 2015. Or even when a USB stick is inserted. Systems have 12 cores now and people can be very sensitive to boot times.*
Parallelism is but a small part of the complicated topic of boot time optimization. And yes, sequential strategies can still be really fast. Much of the speed increase in modern systems is not from parallelism, but forfeiting the overhead of the shell interpreter in favor of direct execution. Checkpointing can do far greater wonders for your boot times than parallelism can, anyhow.
I understand that systemd relies heavily on cgroups and other Linux-specific features for process isolation.
That's only scratching the surface of Linux-specific features in systemd.
As it is, I would be amazed if at least one or two of the BSDs didn't implement a similar init system to systemd in the next few years. And as it is, they probably won't share any code or concepts, unfortunately.
What will come of the BSDs is uncertain. launchd will be used in iXsystems products like PC-BSD and FreeNAS, and there's not much "code sharing" you could do there - it's already a complete product being ported.
Parallelism is but a small part of the complicated topic of boot time optimization. And yes, sequential strategies can still be really fast. Much of the speed increase in modern systems is not from parallelism, but forfeiting the overhead of the shell interpreter in favor of direct execution. Checkpointing can do far greater wonders for your boot times than parallelism can, anyhow.
No. I have worked on embedded Linux systems where we have carefully charted boot time. We were able to get order-of-magnitude improvements by parallelizing service startups during boot. Parallelism is far from "a small part"-- it is the only important part. The overhead of the shell is negligible even on the slowest systems. Checkpointing is not an option for most things (I have mentioned this several times) because most of the startup processes initialize the hardware directly or indirectly, and hardware can't be checkpointed.
The idea of device units is not well borne out, as I mentioned. You don't need to create an object type out of a udev handle to order after devices or trigger hotplug events (the device node manager itself does fine, of course in the case of systemd, udev is part of the project, so...)
The important question is whether there is more or less complexity in systemd than a system with a separate device node manager. I am not convinced that systemd is more complex than the alternative. If the device node manager was separate, you would have to debug the interaction between the two processes. Having dealt with debugging d-bus interactions on Linux before, it doesn't sound fun.
I'm not convinced that a static dependency graph is even feasible in a world where users can reconfigure and install new services, hotplug hardware, and join and leave wireless networks.
No. I have worked on embedded Linux systems where we have carefully charted boot time. We were able to get order-of-magnitude improvements by parallelizing service startups during boot. Parallelism is far from "a small part"-- it is the only important part. The overhead of the shell is negligible even on the slowest systems. Checkpointing is not an option for most things (I have mentioned this several times) because most of the startup processes initialize the hardware directly or indirectly, and hardware can't be checkpointed.
The idea of device units is not well borne out, as I mentioned. You don't need to create an object type out of a udev handle to order after devices or trigger hotplug events (the device node manager itself does fine, of course in the case of systemd, udev is part of the project, so...)
The important question is whether there is more or less complexity in systemd than a system with a separate device node manager. I am not convinced that systemd is more complex than the alternative. If the device node manager was separate, you would have to debug the interaction between the two processes. Having dealt with debugging d-bus interactions on Linux before, it doesn't sound fun.
I'm not convinced that a static dependency graph is even feasible in a world where users can reconfigure and install new services, hotplug hardware, and join and leave wireless networks.
There already are systems for the BSDs that can import systemd units; and have been for more than a couple of years now.
Thanks. As a developer, that is useful to know.
> The terminology may be different, but ultimately upstart's "start on" directive establishes a dependency.
Actually, it is part of the upstart design that its event-driven architecture is not structured in terms of dependencies.
* http://upstart.ubuntu.com/cookbook/#upstart-s-design-why-it-...
Actually, it is part of the upstart design that its event-driven architecture is not structured in terms of dependencies.
* http://upstart.ubuntu.com/cookbook/#upstart-s-design-why-it-...
At the end of the day, this is an issue of terminology. You can say "service X requires service Y to start" or you can say "once service Y has started, start service X." In both cases you can draw a dependency graph with an edge going from Y to X.
Only if one has completely missed the point of what the Cookbook explains.
My take-away is the following:
* systemd tries to solve a complicated problem
* systemd has a lot of complexity
* some of that complexity seems unnecessary
* parsing is hard (see DJB). Parsing (as systemd does) in PID 1 is bad. Complex systems in PID 1 is bad.
* systemd manages states and dependencies (this is good)
* some of the states are only internal, and can't be serialized to storage. The result is indeterminism and race conditions. (This is bad)
* as a result of the above point, systemd behavior is hard to debug, and hard to understand
* another result (not stated explicitly) is that systemd allows for (and apparently can't detect) ordering cycles and dependency cycles.
* The configuration files are simple for simple tasks (good), but not complex enough for complex tasks (bad)
... etc.
i.e. systemd is an engineering solution. It pretty much works, but it has inherent design / implementation issues which cause real-world problems.
* systemd tries to solve a complicated problem
* systemd has a lot of complexity
* some of that complexity seems unnecessary
* parsing is hard (see DJB). Parsing (as systemd does) in PID 1 is bad. Complex systems in PID 1 is bad.
* systemd manages states and dependencies (this is good)
* some of the states are only internal, and can't be serialized to storage. The result is indeterminism and race conditions. (This is bad)
* as a result of the above point, systemd behavior is hard to debug, and hard to understand
* another result (not stated explicitly) is that systemd allows for (and apparently can't detect) ordering cycles and dependency cycles.
* The configuration files are simple for simple tasks (good), but not complex enough for complex tasks (bad)
... etc.
i.e. systemd is an engineering solution. It pretty much works, but it has inherent design / implementation issues which cause real-world problems.
This article pretty conclusively convinced me that systemd is actually good, if these are the strongest arguments against it. Most of the dependency-related stuff (process order related, cyclicstuff) seems eminently fixable. I found his arguments about dependency management having to be linearizable highly unconvincing.
> I found his arguments about dependency management having to be linearizable highly unconvincing.
What? Can you expand on this? To me it rang as a well-reasoned, and pretty devastating, critique.
What? Can you expand on this? To me it rang as a well-reasoned, and pretty devastating, critique.
These are the strongest argument against its core service management logic; those that explain why it is not better than Upstart (or daemontools or s6+s6-rc …)
And systemd causes problems not only by this, but also by its bundling of similarly mediocre or worse other components.
And systemd causes problems not only by this, but also by its bundling of similarly mediocre or worse other components.
"a controversial topic"
My observation: The "discussion" never seems to focus on systemd. Instead it usually turns to comments on the former sysvinit system or other init systems.
That avoidance is perhaps something to ponder. And maybe it's why this author felt the urge to put some focus on systemd itself.
Love the quote from djb. In sum, the best interface is no interface. Parsing amounts to high margin for error and often an incredible nuisance.
In another thread today I wrote about the command line interface and "expecting a reponse". Truthfully, djb's utilities that simply return an exit value and no output are the best ones I have ever used.
(It seems djb himself is a systemd user. Not sure what if anything that means.)
Imagine if the information dissiminated via www was as easy to "parse" as text0. Writing a simple "web browser" might be easy enough that programmers would not need to be paid to do it.
My observation: The "discussion" never seems to focus on systemd. Instead it usually turns to comments on the former sysvinit system or other init systems.
That avoidance is perhaps something to ponder. And maybe it's why this author felt the urge to put some focus on systemd itself.
Love the quote from djb. In sum, the best interface is no interface. Parsing amounts to high margin for error and often an incredible nuisance.
In another thread today I wrote about the command line interface and "expecting a reponse". Truthfully, djb's utilities that simply return an exit value and no output are the best ones I have ever used.
(It seems djb himself is a systemd user. Not sure what if anything that means.)
Imagine if the information dissiminated via www was as easy to "parse" as text0. Writing a simple "web browser" might be easy enough that programmers would not need to be paid to do it.
> The "discussion" never seems to focus on systemd
All the more reason to try making this one better.
(We detached this comment from https://news.ycombinator.com/item?id=10370446 and marked it off-topic.)
All the more reason to try making this one better.
(We detached this comment from https://news.ycombinator.com/item?id=10370446 and marked it off-topic.)
https://news.ycombinator.com/newsguidelines.html