Broken by design: systemd(ewontfix.com)
ewontfix.com
Broken by design: systemd
http://ewontfix.com/14/
11 comments
I've used systemd on Arch Linux for about a year now. It wasn't the end of the world, I can deal with it, but it's not fun.
I'm bothered by seemingly built-in units that have no corresponding unit files, which all default to enabled, including units for bootsplash stuff ("plymouth") that isn't used on arch linux. I'm very annoyed by mounts and such that don't even have units, files or otherwise, such as the "pstore" filesystem or autofs4. I'm super annoyed that since an upgrade half a year ago, the configuration option to disable use of some cgroup controllers was removed, and now any cgroup controllers supported by the kernel are used unconditionally.
I have good reasons for not using some of these things, but even more than that, one of the main reasons I like to use Arch Linux is that I like to add what I need, rather than try to make sense of and clean up the clutter of a typical operating system later. The lack of control and default-everything-on of systemd is really very annoying, and I'm bothered by the strategies mentioned in this article by which systemd makes itself the one and only practical modern init system for linux.
I'm bothered by seemingly built-in units that have no corresponding unit files, which all default to enabled, including units for bootsplash stuff ("plymouth") that isn't used on arch linux. I'm very annoyed by mounts and such that don't even have units, files or otherwise, such as the "pstore" filesystem or autofs4. I'm super annoyed that since an upgrade half a year ago, the configuration option to disable use of some cgroup controllers was removed, and now any cgroup controllers supported by the kernel are used unconditionally.
I have good reasons for not using some of these things, but even more than that, one of the main reasons I like to use Arch Linux is that I like to add what I need, rather than try to make sense of and clean up the clutter of a typical operating system later. The lack of control and default-everything-on of systemd is really very annoying, and I'm bothered by the strategies mentioned in this article by which systemd makes itself the one and only practical modern init system for linux.
I've heard good things about Runit + Monit together: https://gist.github.com/mguterl/279082
I'd like to see Runit [1] and S6 [2] get more attention.
Docs and easy-to-apply tutorials would be a godsend in this area. I too really want to replace systemd on Arch. Currently, this is an uphill battle due to so many packages having systemd service files, and the fact that I've never replaced an init system before makes things that much more daunting.
If only there were more information out there, or more ready-made solutions [3], I'm sure there are plenty who would be ready and willing to give this a shot.
1: http://smarden.org/runit/
2: http://www.skarnet.org/software/s6/
3: https://github.com/chneukirchen/ignite
3: https://github.com/rubyists/runit-services
I'd like to see Runit [1] and S6 [2] get more attention.
Docs and easy-to-apply tutorials would be a godsend in this area. I too really want to replace systemd on Arch. Currently, this is an uphill battle due to so many packages having systemd service files, and the fact that I've never replaced an init system before makes things that much more daunting.
If only there were more information out there, or more ready-made solutions [3], I'm sure there are plenty who would be ready and willing to give this a shot.
1: http://smarden.org/runit/
2: http://www.skarnet.org/software/s6/
3: https://github.com/chneukirchen/ignite
3: https://github.com/rubyists/runit-services
The only Plymouth-specific code I see in systemd is related to asking the user questions at boot-up time, like their mount password, and it works by sending a socket message to a socket plymouth opens, and successfully falls back if it doesn't exist.
I don't see any plymouth-related services or unit files hardcoded. I do, however, see a NEWS-related entry from the v186 release: http://cgit.freedesktop.org/systemd/systemd/tree/NEWS#n1775
I don't see any plymouth-related services or unit files hardcoded. I do, however, see a NEWS-related entry from the v186 release: http://cgit.freedesktop.org/systemd/systemd/tree/NEWS#n1775
there are no unit files, but there are units:
[plo@plo-air ~]$ systemctl --all | grep plym
plymouth-quit-wait.service masked inactive dead plymouth-quit-wait.service
plymouth-start.service masked inactive dead plymouth-start.service
[plo@plo-air ~]$ systemctl --all | grep plym
plymouth-quit-wait.service masked inactive dead plymouth-quit-wait.service
plymouth-start.service masked inactive dead plymouth-start.service
On my system, those units do have unit files, and they're provided by the Plymouth package. I couldn't find any reference to "plymouth-quit-wait" in the systemd source code, so those strings have to be coming from somewhere. Are you sure you don't have a unit file at /usr/lib/systemd/systemd/plymouth-quit-wait.service ?
It seems to me that there is concerted effort to discredit alternative init implementations.
Example: https://en.wikipedia.org/w/index.php?title=Daemontools&actio...
I can't imagine what agency with a history of undermining technical standards could possibly be involved.
Example: https://en.wikipedia.org/w/index.php?title=Daemontools&actio...
I can't imagine what agency with a history of undermining technical standards could possibly be involved.
Why don't the anti-systemd folks publish some actual attacks against that big ol' attack surface?
I've been using it for maybe 18 months, it was different and different is some times scary but I have had no problems at all. Migrated some legacy init scripts and it's a piece of cake. It's fast, it works, it does what it claims to do. I've been shipping unix software mostly on linux for 15 years and there are very real reasons systemd, launchd, smf, upstart and other "do more than nothing" init systems exist and companies making money selling them write them. All of them are much much more simple for the administrator to use too...
I've been using it for maybe 18 months, it was different and different is some times scary but I have had no problems at all. Migrated some legacy init scripts and it's a piece of cake. It's fast, it works, it does what it claims to do. I've been shipping unix software mostly on linux for 15 years and there are very real reasons systemd, launchd, smf, upstart and other "do more than nothing" init systems exist and companies making money selling them write them. All of them are much much more simple for the administrator to use too...
> anti-systemd folks
[note: no personal hostility intended - I'm responding to the idea of "anti-systemd" in the abstract]
It's this kind of us-vs-the, faction-oriented attitude that concerns some of us. Nobody[1] is saying systemd should be expunged from the repositories or removed as an option.
The concern is that a newcomer (i.e. potentially buggy, but could prove itself over time) to the area of "init software" is demanding to be the only software allowed for that particular problem. Even more concerning is how that same newcomer as monopolistic tendencies, and has started tying/bundling together several neighboring software niches, demanding that the currently used and working solutions in those areas be abandoned as well, because of some made up requirement.
If you or anybody else wants to run systemd, you should. In reading these discussion on it, it even sounds like a great choice for some situations (the people setting up "tens of thousands" of mostly-identical systems seem to like it a lot). Some distros make it the default? Why not, variety is one of the great things about having multiple distros to choose from.
Insistence that OpenRC and such must make way for systemd is enough provoke a rather strong reaction. The deliberate incompatibilities to force adoption, on the other hand, are borderline fighting words.
This pattern - trying to suppress competing ideas while claiming the opposition is being unreasonable for suggesting some sort of compromise - is something I've had to fight in politics for a long time. It's expected any time atheism or gay marriage is brought up.
I never thought I would see that kind of "argument" used seriously[2] over UNIX...
1: "nobody", modulo the rare fanatic/ass that you find in any sufficiently large group of people. Even the linked article above - despite the presence of strong opinions - is more concerned with technical issues and how to solve the "init system problem" in general.
2: we have had our Holy Wars, but traditionally only leads to strange stuff like "evil-mode"
[note: no personal hostility intended - I'm responding to the idea of "anti-systemd" in the abstract]
It's this kind of us-vs-the, faction-oriented attitude that concerns some of us. Nobody[1] is saying systemd should be expunged from the repositories or removed as an option.
The concern is that a newcomer (i.e. potentially buggy, but could prove itself over time) to the area of "init software" is demanding to be the only software allowed for that particular problem. Even more concerning is how that same newcomer as monopolistic tendencies, and has started tying/bundling together several neighboring software niches, demanding that the currently used and working solutions in those areas be abandoned as well, because of some made up requirement.
If you or anybody else wants to run systemd, you should. In reading these discussion on it, it even sounds like a great choice for some situations (the people setting up "tens of thousands" of mostly-identical systems seem to like it a lot). Some distros make it the default? Why not, variety is one of the great things about having multiple distros to choose from.
Insistence that OpenRC and such must make way for systemd is enough provoke a rather strong reaction. The deliberate incompatibilities to force adoption, on the other hand, are borderline fighting words.
This pattern - trying to suppress competing ideas while claiming the opposition is being unreasonable for suggesting some sort of compromise - is something I've had to fight in politics for a long time. It's expected any time atheism or gay marriage is brought up.
I never thought I would see that kind of "argument" used seriously[2] over UNIX...
1: "nobody", modulo the rare fanatic/ass that you find in any sufficiently large group of people. Even the linked article above - despite the presence of strong opinions - is more concerned with technical issues and how to solve the "init system problem" in general.
2: we have had our Holy Wars, but traditionally only leads to strange stuff like "evil-mode"
Folks on #musl found 2 integer overflow bugs in the UTF-8 handling code within seconds of checking out the source to read it. It's not clear to me where the code is called from and what inputs reach it, but that kind of bug does not inspire confidence in the safety of the project as a whole. I also quickly found a case where a potentially-null string pointer is passed to a logging function which might or might not be able to accept null string pointers. (I suspect it uses snprintf and therefore depends on the underlying C library's handling of this usage, which is undefined behavior.)
In the interest of transparency, can you tell us exactly what you found? I looked around for a bit but couldn't spot anything.
The argument is one of minimizing attack surface, not specific current attacks. I.e. it's a question of selecting the right technology security-wise and attack surface is a big part of that equation because with that surface area comes the strong likelihood that attacks will be found against the process over time.
I use systemd because that's what my distro comes with. There are positives and negatives. On the whole though, I agree with much of the criticism, and I hope that a real alternative comes along which is more security and uptime friendly.
I use systemd because that's what my distro comes with. There are positives and negatives. On the whole though, I agree with much of the criticism, and I hope that a real alternative comes along which is more security and uptime friendly.
How has it affected your uptime? Any specifics?
In addition to the prudential concern issues (which are laid out in the article), there are some concrete issues:
With sysv init, it is very easy to add new services that may not be supported out of the box. There are scripts like rc.local, etc, which can be edited to start anything. This means that setting up a new service that can be supported properly is a lot more work, especially if it is for development systems. While I haven't had systemd crash, I have had issues with stuff requiring manual start (i.e logging in as root and running the startup script for the app) when a simple rc.local-type approach would have been far simpler.
Another problem I have had is the issue of process accounting, where if the process starts up a daemon and exits, then systemd has trouble determining the status properly. This can cause issues for things like monitoring (which means basically I don't use systemd for monitoring whether services are running).
What this means, in essence, is that to make this work the way I would like I have had to reinvent part of the sysv init system inside of systemd and then handle things like process tracking the old fashioned way (pid files and shell scripts).
In short, in addition to the "I don't want my system to have to reboot and this is a bad architecture for that requirement" you have quite a bit of areas where there are leaky abstraction layers which don't really add any benefit to administration of the system.
With sysv init, it is very easy to add new services that may not be supported out of the box. There are scripts like rc.local, etc, which can be edited to start anything. This means that setting up a new service that can be supported properly is a lot more work, especially if it is for development systems. While I haven't had systemd crash, I have had issues with stuff requiring manual start (i.e logging in as root and running the startup script for the app) when a simple rc.local-type approach would have been far simpler.
Another problem I have had is the issue of process accounting, where if the process starts up a daemon and exits, then systemd has trouble determining the status properly. This can cause issues for things like monitoring (which means basically I don't use systemd for monitoring whether services are running).
What this means, in essence, is that to make this work the way I would like I have had to reinvent part of the sysv init system inside of systemd and then handle things like process tracking the old fashioned way (pid files and shell scripts).
In short, in addition to the "I don't want my system to have to reboot and this is a bad architecture for that requirement" you have quite a bit of areas where there are leaky abstraction layers which don't really add any benefit to administration of the system.
That sounds strangely like the problem systemd was created to solve. It's supposed to detect double forks in the started daemon and track pid files named in it's configuration file.
At least that's how I read it in numerous articles on LWN and mailing lists, I haven't tried it on an actual system yet.
At least that's how I read it in numerous articles on LWN and mailing lists, I haven't tried it on an actual system yet.
The maintainer of mdadm migrated two nfs-utils sysvinit scripts of 801 lines into 14 systemd unit files of 168 lines.[1] Complexity won't go away with migration.
[1] http://lwn.net/Articles/584175/ (Subscriber only)
[1] http://lwn.net/Articles/584175/ (Subscriber only)
I believe that people will start pushing attacks once the popularity of sytemd goes up. Why attack something that's not as widely used as sysvinit?
I don't think that the argument was that it was scary to upgrade or there were no benefits; the author was specifically stating that a couple of points inherent to the design of systemd make it something that he finds less stable. Systemd could still provide these benefits by shipping more than one binary for extra features, reducing the attack surface of the init system itself.
EDIT: BTW I use archlinux and therefore use systemd.
I don't think that the argument was that it was scary to upgrade or there were no benefits; the author was specifically stating that a couple of points inherent to the design of systemd make it something that he finds less stable. Systemd could still provide these benefits by shipping more than one binary for extra features, reducing the attack surface of the init system itself.
EDIT: BTW I use archlinux and therefore use systemd.
> systemd makes it impossible to upgrade without rebooting. This leads to "Linux" becoming the laughing stock of Windows fans, as happened with Ubuntu a long time ago.
This irked me. It's just an unnecessary comparison. The ZDnet article linked is, and always was, completely fallacious in comparing an OS critical update channel to a fully-fledged distro package manager. Having just bought a brand new Windows laptop this weekend, I found myself spending no less than 2 hours tirelessly updating and rebooting Windows, and searching for, and installing, driver updates manually. It's 2014 and it still sucks. Windows update and OEM updaters are, in my experience, piss-poor and don't pick up half of the updates available to your average system.
Frankly I think your average Linux desktop could easily risk doubling or tripling the number and frequency of reboots demanded, and you'd still not reach a significant fraction of the annoyance of Windows on that particular front.
This irked me. It's just an unnecessary comparison. The ZDnet article linked is, and always was, completely fallacious in comparing an OS critical update channel to a fully-fledged distro package manager. Having just bought a brand new Windows laptop this weekend, I found myself spending no less than 2 hours tirelessly updating and rebooting Windows, and searching for, and installing, driver updates manually. It's 2014 and it still sucks. Windows update and OEM updaters are, in my experience, piss-poor and don't pick up half of the updates available to your average system.
Frankly I think your average Linux desktop could easily risk doubling or tripling the number and frequency of reboots demanded, and you'd still not reach a significant fraction of the annoyance of Windows on that particular front.
And all the servers? They are supposed to just keep rebooting all the time as if they were Windows desktops?
If you upgrade systemd or the kernel, yes, you should reboot your machine. Ideally, it shouldn't take more than a minute or two.
Obviously not talking about servers here. Those RAID can take more than a few minutes to spin-up.
If you can't afford 3 minutes of downtime (how long a Dell PowerEdge R620 takes to start responding to ping again after a shutdown -r now), you really should reconsider running that service without a redundant system at the ready.
It seems to me that, in keeping with its long tradition of being conservative, Debian should put off switching its init system for at least another release, since systemd is by no means the clear winner from a strictly technical point of view.
Agreed here. Then, at that time, there should be a re-evaluation.
Perhaps also at that time openRC will be a potential candidate and far less experimental.
Perhaps also at that time openRC will be a potential candidate and far less experimental.
it seems like the pid 1 issue has been discussed quite a few times in the past. see also here https://lwn.net/Articles/520892/
it's also discussed in the official debian debate https://wiki.debian.org/Debate/initsystem/systemd
can anyone explain to me why they haven't just changed systemd to do less work in pid 1?
i've used systemd on my archlinux laptop for quite a while and was happy with how much faster the system boots. but i also found it to be too huge and too intertwined for basic maintenance tasks. but ofc that is just a personal opinion. suddenly i have to have a repertoire of systemd commands to do things i used to be able to do with basic unix tools
it's also discussed in the official debian debate https://wiki.debian.org/Debate/initsystem/systemd
can anyone explain to me why they haven't just changed systemd to do less work in pid 1?
i've used systemd on my archlinux laptop for quite a while and was happy with how much faster the system boots. but i also found it to be too huge and too intertwined for basic maintenance tasks. but ofc that is just a personal opinion. suddenly i have to have a repertoire of systemd commands to do things i used to be able to do with basic unix tools
I don't know. Given what I know about how cgroups and systemd works, I don't think there's any particular issue with having a thin shim and starting systemd as PID 2, and so getting some of the reliability gains this article talks about. But I don't have all that much knowledge of the details and there may be good technical reasons why that is a bad idea (the man page notes that it is possible via a command line flag, but that it is not supported and only intended for testing). There are some issues surrounding cgroups moving to a single-userspace-arbitrator model, but I don't know what requires this to be PID 1.
The article remarks its pid 1 to catch "bad" services that disown their parents and get defaulted to pid 1.
But I figure if they switched the bulk of systemd to pid 2, any negligent services would get complained about until they were fixed.
But I figure if they switched the bulk of systemd to pid 2, any negligent services would get complained about until they were fixed.
Almost all daemons have a command line or config file option to skip "daemonizing" so that the parent can watch the PID correctly. For the few that don't, you can fix them, hack them (e.g. LD_PRELOAD wrappers to shutdown their bogus double-forking), seccomp-ptrace them, or simply wrap them with a legacy-style monitor that reads the pid file and forwards signals it receives to the real pid (this last option of course doesn't fix the race conditions inherent in such usage).
This is something systemd can already handle. Linux has APIs to change the 'reaper' process (PR_SET_CHILD_SUBREAPER) so that orphaned processes move to a non PID 1 process.
Was openrc ever a part of the discussion?
I don't like the concept of a GUI/Redhats philosophy dictating system behavior when X isn't even installed.
I don't like the concept of a GUI/Redhats philosophy dictating system behavior when X isn't even installed.
I like this writing. Solid.
But, I'm curious: What are the specific deficiencies the author sees in daemontools? "Not ready for primetime."
I mean, it's so typical for bloggers to say certain software is this or that (some generalisation), without giving details.
I don't know much about systemd. I find Linux too chaotic, a moving target that moves a little too frequently for my tastes. It's changes like these that I find take too much energy to follow.
I doubt it's a problem specific to Linux though. As a BSD user, I've been amazed with how much init.c has grown in size over the years. A variety of developers have touched it.
But, I'm curious: What are the specific deficiencies the author sees in daemontools? "Not ready for primetime."
I mean, it's so typical for bloggers to say certain software is this or that (some generalisation), without giving details.
I don't know much about systemd. I find Linux too chaotic, a moving target that moves a little too frequently for my tastes. It's changes like these that I find take too much energy to follow.
I doubt it's a problem specific to Linux though. As a BSD user, I've been amazed with how much init.c has grown in size over the years. A variety of developers have touched it.
I'll run systemd only after you pry openrc[1] from my cold, dead hands.
FAR too much of the project makes exactly the wrong design choices. The worst of these flaws is the binding of previously-independent components discussed near the end of this article.
This bundling Is not even necessary! The features that systemd gets right - an interesting daemon management tool, and providing a useful default start/stop mechanism ("4 lines of config instead of re-writing the same generic shell script") - can obviously be independent features.
This mess should be limited to merely bad design in a particular software niche(s)., but now we're seeing the "my way or nothing" attitude spill over into existing user-level software. (e.g., Gnome[2] I would probably already be using part of it was properly modular. Instead, the bundle is growing even more monolithic and incompatible.
When a similar "forced upgrade/incompatibility" happened to OpenGL (with ES dropping the traditional fixed-function pipeline, JWZ had a very good summary[3] of the the problem, from a "having to maintain old software" prespective:
1: Not giving up my .xinitrc/.Xresources or E16 theme, either. In all cases (along with openrc/sysvinit), they have over a decade's worth of bugfixes and tweaks. Any replacement would have to be very good to be worth the time/effort involved in switching.
2: to be fair, Gnome has been merging thing that shouldn't be merged all the way back to the beginning of the project, and is as much at fault here as systemd itself.
3: http://www.jwz.org/blog/2012/06/i-have-ported-xscreensaver-t...
FAR too much of the project makes exactly the wrong design choices. The worst of these flaws is the binding of previously-independent components discussed near the end of this article.
This bundling Is not even necessary! The features that systemd gets right - an interesting daemon management tool, and providing a useful default start/stop mechanism ("4 lines of config instead of re-writing the same generic shell script") - can obviously be independent features.
This mess should be limited to merely bad design in a particular software niche(s)., but now we're seeing the "my way or nothing" attitude spill over into existing user-level software. (e.g., Gnome[2] I would probably already be using part of it was properly modular. Instead, the bundle is growing even more monolithic and incompatible.
When a similar "forced upgrade/incompatibility" happened to OpenGL (with ES dropping the traditional fixed-function pipeline, JWZ had a very good summary[3] of the the problem, from a "having to maintain old software" prespective:
Let's say you have a well-specified system that is in wide use (a language, a
library API, whatever) and because of changes in some substrate (operating
systems, hardware, whatever) you find that you need to add a new way of doing
things to it.
The way you do this is, you add new features to the specification and you
clearly document the version in which those features become supported.
If there are old features that you would like to discourage the use of, then
you mark them as obsolete -- but you do not remove them because thou shalt not
break working code.
If you don't agree with that, then please, get out of the software industry
right now. Find another line of work. Please.
Your users have code that works. Maybe the new APIs would serve them better.
Maybe things would be so much more efficient if they updated their code to use
the new API. Or maybe it doesn't matter to them and they just want working code
to continue to be working code. At least until such a time as they need the new
features, or new efficiency. Remember the First Rule of Optimization: DON'T.
1: Not giving up my .xinitrc/.Xresources or E16 theme, either. In all cases (along with openrc/sysvinit), they have over a decade's worth of bugfixes and tweaks. Any replacement would have to be very good to be worth the time/effort involved in switching.
2: to be fair, Gnome has been merging thing that shouldn't be merged all the way back to the beginning of the project, and is as much at fault here as systemd itself.
3: http://www.jwz.org/blog/2012/06/i-have-ported-xscreensaver-t...
I think Gnome has been taking that approach going back years in part because of the opposite criticism: many people attack Linux UI as being a hodge-podge with no coherent design, comparing it unfavorably to a unified UI system like OSX's, which has a coherent and integrated take on windowing/libraries/widgets/etc. One of Gnome's goals was to fix that. They might unsuccessful at doing so, but attempting a unified/integrated UI was/is a core project goal, so I'm not surprised that they'd have no shyness about integrating components that Unix has traditionally kept separate. The tiling window manager scene (which I prefer myself) is perhaps the polar opposite (some wms don't even integrate the status bar into the wm), but isn't really targeting Gnome's target audience either.
systemd is like a second kernel in user-space.
Which should be good. If the OPs arguments about how dangerous it is to empower and stuff a lot of functionality into pid 1, think of how running in kernel mode on a cpu gives you power when your code is in the kernel proper.
So moving functionality out of kernel mode and into user mode is great. The only steps after that are moving as much functionality as possible to the user, and then to nobody. We do trend towards minimum permissions for the sake of security for a reason.
So moving functionality out of kernel mode and into user mode is great. The only steps after that are moving as much functionality as possible to the user, and then to nobody. We do trend towards minimum permissions for the sake of security for a reason.
Functionality is not being moved out of the kernel and into userspace. It's being moved out of legacy freedesktop.org components (hald, ConsoleKit, PolicyKit, etc.) and into PID 1. Where should it be moved to? /dev/null.
I think you need to look a bit more closely at what's actually going into PID 1. See for example: http://people.debian.org/~stapelberg/docs/systemd-dependenci...
I don't see any logind/hostnamed/journald or any of that stuff in PID1.
I don't see any logind/hostnamed/journald or any of that stuff in PID1.
hald has been dead forever.
PolicyKit is not being moved into systemd, nor is it getting systemd integration?
How do you want to replace ConsoleKit/logind? What should handle session management?
PolicyKit is not being moved into systemd, nor is it getting systemd integration?
How do you want to replace ConsoleKit/logind? What should handle session management?
and,
launchd is like a second mach kernel at a higher level. https://developer.apple.com/library/mac/documentation/Darwin...
Except for the part where launchd doesn't implement anything other than traditional process/service launch and termination, and doesn't violate the layering boundaries between system components.
In a launchd system, there's absolutely no dependency on launchd by userspace software.
In a launchd system, there's absolutely no dependency on launchd by userspace software.
Just to ensure we're all talking about the same thing, here's an overview of systemd's footprint: http://people.debian.org/~stapelberg/docs/systemd-dependenci...
From the article: "On most machines, systemd will bring from 236 KiB to 550 KiB of additional libraries to be loaded into memory. When recompiling systemd without its optional dependencies (e.g. for embedded systems), this shrinks to 368 KiB in the worst case.
The RSS (Resident Set Size) of systemd is 1.8 MiB, whereas sysvinit uses 0.8 MiB — both measured on typical Debian wheezy installations."