What should we expect from a modern init system?(plus.google.com)
plus.google.com
What should we expect from a modern init system?
https://plus.google.com/u/0/+LennartPoetteringTheOneAndOnly/posts/ip8e1DqJdxT
97 comments
* Sh is a terrible language to build any non-trivial functionality.
* System initialization has become more complex.
* Currently, managing daemons has a lot boilerplate code.
* That code is sub-optimal and fragile.
Personally, I don't want to know anything about the timing on which each device is available, to figure out manually the right order to start the services. That's the kind of job a program can and should do. I didn't like when I have to set IRQs manually too, or when X asked me about the data of the video chip that was connected to its bus.
Back to the init scripts. How do those scripts know if a daemon is still running? One mechanism is to check the existence of a file with the daemon's pid! IMO the current system is an anti-aesthetic patch. Only inertia keeps it alive.
This is what I would like to see in a linux system:
* A common config language for the basic services. Something like the defunct [elektra initiative](http://freedesktop.org/wiki/Software/Elektra/) * An automate mechanism to orderly start the system. * Something like Dan Berstein's daemontool to handle the daemons, plus the cron jobs. * For Ritchie's sake, keep logs as text files!
Personally, I don't want to know anything about the timing on which each device is available, to figure out manually the right order to start the services. That's the kind of job a program can and should do. I didn't like when I have to set IRQs manually too, or when X asked me about the data of the video chip that was connected to its bus.
Back to the init scripts. How do those scripts know if a daemon is still running? One mechanism is to check the existence of a file with the daemon's pid! IMO the current system is an anti-aesthetic patch. Only inertia keeps it alive.
This is what I would like to see in a linux system:
* A common config language for the basic services. Something like the defunct [elektra initiative](http://freedesktop.org/wiki/Software/Elektra/) * An automate mechanism to orderly start the system. * Something like Dan Berstein's daemontool to handle the daemons, plus the cron jobs. * For Ritchie's sake, keep logs as text files!
daemontools has the best philosophy for this in my opinion: just start the service. If the prerequisites are unavailable, the service will just terminate. If that happens, wait one second, then try start again.
It's perfect. No stupid 5 minute timeout like sysvinit, no stupid dependency management like systemd/upstart, very little to consider when adding new services.
If you want to use it as your init.. just look into 'runit-init'. I've been using this for 4+ years, and it's fantastic.
It's perfect. No stupid 5 minute timeout like sysvinit, no stupid dependency management like systemd/upstart, very little to consider when adding new services.
If you want to use it as your init.. just look into 'runit-init'. I've been using this for 4+ years, and it's fantastic.
My LG G2, running android 4.2.2 with a 3.2.0 kernel, boots in about eight seconds from cold. It also has the best battery life of any smartphone I've ever seen (24+ hours, of which 8+ with screen on and doing things is normal on any day that I don't play 3D games.) SysVinit.
My boring house server, AMD FX4130, boots from a small Intel SSD in forty seconds from power to 0 load. Debian stable, sysVinit.
I understand sysVinit. I can debug it. When things go wrong, I can stop it, figure it out, fix it and start it again.
Being able to figure things out and fix them is the heart of UNIX. I'm not against progress, I'm against screwing things up in the name of progress.
OpenRC looks interesting.
My boring house server, AMD FX4130, boots from a small Intel SSD in forty seconds from power to 0 load. Debian stable, sysVinit.
I understand sysVinit. I can debug it. When things go wrong, I can stop it, figure it out, fix it and start it again.
Being able to figure things out and fix them is the heart of UNIX. I'm not against progress, I'm against screwing things up in the name of progress.
OpenRC looks interesting.
> I understand sysVinit. I can debug it. When things go wrong, I can stop it, figure it out, fix it and start it again.
"I don't want to learn new things. I labour under the delusion that tens of thousands of lines of shell code is a clean solution" is all I hear.
"I don't want to learn new things. I labour under the delusion that tens of thousands of lines of shell code is a clean solution" is all I hear.
Then you did not read or failed to understand the line that came after the one you quoted.
You LG2 has probably optimized, vendor/hardware specific scripts, SysVinit does not achieve the same perfs on my high-end laptop.
You may want to have a look at this short conference from linux.conf.au, about systemd from the point of view of a sysadmin: http://www.youtube.com/watch?v=-97qqUHwzGM
You may want to have a look at this short conference from linux.conf.au, about systemd from the point of view of a sysadmin: http://www.youtube.com/watch?v=-97qqUHwzGM
My midrange old laptop (Intel Core2Duo, 1.4GHz, Intel SSD) boots in about thirty seconds from cold to X login with sysVinit. Again, Debian Stable, nothing odd going on.
There are tools to look at what your laptop is doing during boot time.
I've seen the systemd and upstart propaganda, and find it unconvincing.
There are tools to look at what your laptop is doing during boot time.
I've seen the systemd and upstart propaganda, and find it unconvincing.
Propaganda? Jeez.
sysvinit is long overdue retirement - I've worked with it for many years can't wait for it to die.
It can't handle power management, it calls out to a multitude of scripts/external applications just to stop start and restart (oh, and which command options are the correct ones to support?), it can't manage processes through their lifecycle.
People constantly trot out that it's not the unix way... Linux is not unix. X is going away next, are you going to fight to keep that all the way too?
Also, why use external tools to analyse the bootup? $ systemd-analyze Startup finished in 785ms (kernel) + 395ms (initrd) + 2.184s (userspace) = 3.366s $ systemd-analyze plot > file.svg
sysvinit is long overdue retirement - I've worked with it for many years can't wait for it to die.
It can't handle power management, it calls out to a multitude of scripts/external applications just to stop start and restart (oh, and which command options are the correct ones to support?), it can't manage processes through their lifecycle.
People constantly trot out that it's not the unix way... Linux is not unix. X is going away next, are you going to fight to keep that all the way too?
Also, why use external tools to analyse the bootup? $ systemd-analyze Startup finished in 785ms (kernel) + 395ms (initrd) + 2.184s (userspace) = 3.366s $ systemd-analyze plot > file.svg
My android (Huawei G500-0100) boots in 1.5 minutes. Does that mean that we have to ditch OpenRC or whatever init system it has?
The post linked describes quite well an example situation systemd solves, and why something like systemd is needed. Perhaps you should look at it.
Isn't that an implementation detail though? sysvinit is just a series of scripts that are grouped by runlevel. It says nothing about what those scripts must do; it could very well be that a better implementation of the mounting scripts fixes the problem without having to throw out the baby with the bathwater.
As far as I know, it's just features, features, features: http://0pointer.de/blog/projects/why.html
Have you watched this video? http://www.youtube.com/watch?v=-97qqUHwzGM
It's a good introduction to the things systemd brings to the table and why it's worthwhile.
It's a good introduction to the things systemd brings to the table and why it's worthwhile.
Server hardware init is also a problem, the Coreboot project started because of that (IIRC one admin mentioned something in ~hour for a cluster).
Android does not use sysv init, it uses its own init program (keeping the name, but completely different).
>I still do not understand what problem upstart or systemd are trying to solve
Linux systems are still too much like unix. There are still some simple components that do a single, simple task, and can be easily debugged. This is clearly not acceptable, and does not fit with the rest of a modern linux distro. It needs to do more things, be less reliable, and be harder to figure out when something has gone wrong. Systemd and upstart both solve these problems nicely.
Linux systems are still too much like unix. There are still some simple components that do a single, simple task, and can be easily debugged. This is clearly not acceptable, and does not fit with the rest of a modern linux distro. It needs to do more things, be less reliable, and be harder to figure out when something has gone wrong. Systemd and upstart both solve these problems nicely.
Well.. Lennart points this out, then walks right past it. The problem isn't the init, or the boot process, or any of that. The problem is the reliance on 'fstab'.
This is where Lennart/Freedesktop kill me; rather than notice that the underlying mechanism should be replaced, they figure they should just build a better mousetrap on top of it.
If you want to make linux more "desktopy" without screwing around too much with stupid initd decisions, then fix some of these broken underlying mechanisms.
Unless he's really suggesting that my big problem is that my /usr partition might be on a removable device.
This is where Lennart/Freedesktop kill me; rather than notice that the underlying mechanism should be replaced, they figure they should just build a better mousetrap on top of it.
If you want to make linux more "desktopy" without screwing around too much with stupid initd decisions, then fix some of these broken underlying mechanisms.
Unless he's really suggesting that my big problem is that my /usr partition might be on a removable device.
I expect an init system to be an init system, not to also be pm-utils, (x)inetd, acpid, syslogd, watchdog, cron and atd. I also expect it to have configuration files I can change, not to have everything symlinked to /usr. (If I can even find the right configuration file in that mess of directories, that is.)
> I expect an init system to be an init system, not to also be pm-utils, (x)inetd, acpid, syslogd, watchdog, cron and atd.
The problem with this traditional unix approach is that in addition to init, pm-utils, inetd, cron, etc you end up having thousands of lines of shell scripts that glue all this together in a big brittle mess.
While this approach may be fine for servers which boot once and stay on for ages, are connected to fast wired network and are generally rather stable configurations, I don't want such a system for my desktop, laptop or mobile device.
I expect modern devices to be able to deal with changes in power supply (battery/mains/low battery), connecting and disconnecting devices (storage, peripherals, audio devices, etc) and changes in network connectivity (intentional disconnection as well as network failure). This may involve starting and stopping services, mounting and unmounting partitions and taking other actions when the physical configuration changes.
I also expect my computers to boot and shutdown quickly and effectively.
Some of this is already doable in other systems, like the traditional Debian network setup (originally built in the era of wired networks) which can run scripts on connect/disconnect. But the situation ends up being what I described above: thousands of lines of brittle shell scripts.
I welcome the changes and the effort put in systemd and other init systems. Traditional unix init just doesn't cut it any more, in particular on personal desktop and mobile devices.
The problem with this traditional unix approach is that in addition to init, pm-utils, inetd, cron, etc you end up having thousands of lines of shell scripts that glue all this together in a big brittle mess.
While this approach may be fine for servers which boot once and stay on for ages, are connected to fast wired network and are generally rather stable configurations, I don't want such a system for my desktop, laptop or mobile device.
I expect modern devices to be able to deal with changes in power supply (battery/mains/low battery), connecting and disconnecting devices (storage, peripherals, audio devices, etc) and changes in network connectivity (intentional disconnection as well as network failure). This may involve starting and stopping services, mounting and unmounting partitions and taking other actions when the physical configuration changes.
I also expect my computers to boot and shutdown quickly and effectively.
Some of this is already doable in other systems, like the traditional Debian network setup (originally built in the era of wired networks) which can run scripts on connect/disconnect. But the situation ends up being what I described above: thousands of lines of brittle shell scripts.
I welcome the changes and the effort put in systemd and other init systems. Traditional unix init just doesn't cut it any more, in particular on personal desktop and mobile devices.
I had my reservations when Arch shifted to systemd. It seems like a tremendous break from Unix philosophy and a blatant violation of everything I knew.
I'm glad they did, though. Unit files are far easier to write and having a supervisor behavior in the init system alleviates a tremendous amount of administrative overhead. That's not to say shell scripts are difficult, but they can break in mysterious ways. Though I imagine the same is true for systemd, from an end user perspective, its implementation (once you understand it) requires less effort.
There are those who disagree, and I respect that. But the improvements to my desktop boot time are substantial. And writing a quick unit file to run something on boot requires little mental overhead. Just a glance over the manpage is enough to get started. sysvinit scripts on the other hand... well, maybe my memory is really poor, but if it's been more than a few months since I've written one for the specific system I'm targeting, it takes substantially longer. ;)
So, it's likely I'm stupid or systemd is easier. Or both. Probably both.
I'm glad they did, though. Unit files are far easier to write and having a supervisor behavior in the init system alleviates a tremendous amount of administrative overhead. That's not to say shell scripts are difficult, but they can break in mysterious ways. Though I imagine the same is true for systemd, from an end user perspective, its implementation (once you understand it) requires less effort.
There are those who disagree, and I respect that. But the improvements to my desktop boot time are substantial. And writing a quick unit file to run something on boot requires little mental overhead. Just a glance over the manpage is enough to get started. sysvinit scripts on the other hand... well, maybe my memory is really poor, but if it's been more than a few months since I've written one for the specific system I'm targeting, it takes substantially longer. ;)
So, it's likely I'm stupid or systemd is easier. Or both. Probably both.
I had the same experience, would not have expected to be so pleasantly surprised by an init system :) And writing unit files for systemd was fantastically easy.
systemd actually looks like it was designed based on experience. "Worse is Better" as a development philosophy works for getting stuff up and running and finding out what the real problems are so that you don't make solutions for non-existent problems. But when enough time has passed, and enough glue and pieces of string accumulated, it's time to take a step back and consider a redesign.
systemd actually looks like it was designed based on experience. "Worse is Better" as a development philosophy works for getting stuff up and running and finding out what the real problems are so that you don't make solutions for non-existent problems. But when enough time has passed, and enough glue and pieces of string accumulated, it's time to take a step back and consider a redesign.
I think that's my biggest problem with shell-scripts-as-an-init-system. There's nothing inherently wrong with it, but sooner or later, the only way to correct for broken or missing functionality is (to borrow your term) to add more glue. And if you want more correct behavior, you either have to wait for the distro maintainers to fix it or fix it yourself only to merge your changes back in on the next OS update.
That's another thing I like about systemd. With the separation of "stock" unit files in /usr/lib and user-supplied ones in /etc/systemd, you can override the ones distributed by your OS vendor without worrying about updates breaking your changes. That's already saved me once or twice during Arch's transition from sysvinit to systemd, in fact.
One thing that comes to mind is an issue I had some time back with running a proprietary service in a manner that would allow the init system to control it when it forked itself a number of times and spun up separate processes. start-stop-daemon was confused by this behavior and it required some tricks to collect the approriate PID just to be able to stop the service (I may be wrong, but I think it was the TeamSpeak 3 daemon which behaves weirdly). systemd's use of cgroups ignores this rubbish entirely, because it's effectively impossible for systemd to lose track of what processes belong to which service. Want to kill the service? No problem.
Some in this thread seem convinced that systemd is a solution looking for a problem, but I suspect some of them lack experience with systemd. Or stubbornly cling to their ideals of what an init system should do. I like the unification of a supervisor-type service and the init system. It makes my job easier. And that's not even half of what systemd is capable of!
That's another thing I like about systemd. With the separation of "stock" unit files in /usr/lib and user-supplied ones in /etc/systemd, you can override the ones distributed by your OS vendor without worrying about updates breaking your changes. That's already saved me once or twice during Arch's transition from sysvinit to systemd, in fact.
One thing that comes to mind is an issue I had some time back with running a proprietary service in a manner that would allow the init system to control it when it forked itself a number of times and spun up separate processes. start-stop-daemon was confused by this behavior and it required some tricks to collect the approriate PID just to be able to stop the service (I may be wrong, but I think it was the TeamSpeak 3 daemon which behaves weirdly). systemd's use of cgroups ignores this rubbish entirely, because it's effectively impossible for systemd to lose track of what processes belong to which service. Want to kill the service? No problem.
Some in this thread seem convinced that systemd is a solution looking for a problem, but I suspect some of them lack experience with systemd. Or stubbornly cling to their ideals of what an init system should do. I like the unification of a supervisor-type service and the init system. It makes my job easier. And that's not even half of what systemd is capable of!
This is not the fault of unix, it's well suited for static systems. It's the fault of implementers trying to use something that was designed for an entirely different paradigm.
Future apps will need to copy themselves and their data to other devices, perhaps even suspend on one device and wake-up on another.
Making changes for one use case at the expense of another without knowing the history is an ignorant change for churn's sake. Churn is not productivity, but the appearance of doing something.
Future apps will need to copy themselves and their data to other devices, perhaps even suspend on one device and wake-up on another.
Making changes for one use case at the expense of another without knowing the history is an ignorant change for churn's sake. Churn is not productivity, but the appearance of doing something.
>The problem with this traditional unix approach is that in addition to init, pm-utils, inetd, cron, etc you end up having thousands of lines of shell scripts that glue all this together in a big brittle mess.
No, I don't. Typical linux distros may, but I sure as hell don't. I have a very simple, easy to read and debug set of shell scripts that work very well: http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/etc/rc?...
No, I don't. Typical linux distros may, but I sure as hell don't. I have a very simple, easy to read and debug set of shell scripts that work very well: http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/etc/rc?...
The parent comment captures why systemd makes me uneasy, and why I moved from Arch to Debian when I noticed Arch had replaced init with systemd.
The Art of Unix Programming (the book) suggests that software should do one thing and do it well, with easy, plain-text interfaces to its input and output. Systemd tries to do everything -- replacing by default so many core system components like syslog and cron.
But what really bothered me was when I realized that systemd stores all the system logfiles in binary by default! I couldn't believe it. All the system log files: boot, kern, messages, whatever, were stored in some binary format. At that point, rather than research how this new binary log file was implemented, I abandoned Arch and systemd for the init system that had served me well for 15 years.
Log files don't need to be in binary. It means we can't use our standard tools: cut, sort, grep, awk, etc. It means we can't use scripts we have written built on those tools we may have built over decades of working with UNIX.
I guess my point is, systemd, by storing things in binary format and by trying to be the 'everything' daemon, violates two core UNIX programming philosophies: use plain-text interfaces & do one thing well.
The Art of Unix Programming (the book) suggests that software should do one thing and do it well, with easy, plain-text interfaces to its input and output. Systemd tries to do everything -- replacing by default so many core system components like syslog and cron.
But what really bothered me was when I realized that systemd stores all the system logfiles in binary by default! I couldn't believe it. All the system log files: boot, kern, messages, whatever, were stored in some binary format. At that point, rather than research how this new binary log file was implemented, I abandoned Arch and systemd for the init system that had served me well for 15 years.
Log files don't need to be in binary. It means we can't use our standard tools: cut, sort, grep, awk, etc. It means we can't use scripts we have written built on those tools we may have built over decades of working with UNIX.
I guess my point is, systemd, by storing things in binary format and by trying to be the 'everything' daemon, violates two core UNIX programming philosophies: use plain-text interfaces & do one thing well.
Log files being in binary lets you do things like filter based on specific fields rather than on more complicated string matching. For example, you can get log file messages from postfix's smtp process (as opposed to its smtpd process) with one command (journalctl /usr/lib/postfix/smtp), which you can then cut/sort/grep/awk/etc.
You can also follow the logs with journalctl -f and go from there as well.
The problem that this solves is the classic 'what log file does this daemon log this error level to on this system', which varies a surprising amount between daemons, versions, distributions, etc.
I don't actually care which log file I look at, I just want to know what errors monit is getting. systemd and journalctl provide that handily.
I would agree with an approach to have systemd send log events to e.g. journald (or syslog-ng, or whatever the user wanted) to separate that functionality out, however.
You can also follow the logs with journalctl -f and go from there as well.
The problem that this solves is the classic 'what log file does this daemon log this error level to on this system', which varies a surprising amount between daemons, versions, distributions, etc.
I don't actually care which log file I look at, I just want to know what errors monit is getting. systemd and journalctl provide that handily.
I would agree with an approach to have systemd send log events to e.g. journald (or syslog-ng, or whatever the user wanted) to separate that functionality out, however.
I don't mind journalctl in spite of my initial reservations, and as you point out with -f, it's not all that different from tailing a text log. I do suspect binary logging isn't the only reason hkhanna dumped Arch for Debian. It smells of nitpickery to me.
Using journalctl instead of hunting for the appropriate log files in /var/log seems to offer some degree of platform unity. Maybe it's not the right answer, but it works well enough. My primary complaint is how long it can take on a noisy system to filter a large log, but it's not exceptionally noisome. And most of the time, I just want to follow the log.
> I would agree with an approach to have systemd send log events to e.g. journald (or syslog-ng, or whatever the user wanted) to separate that functionality out, however.
While it probably isn't what you mean as you attach the caveat of task separation, it is possible for syslog-ng to collected logging from systemd's journal. It's a little flaky but it works. :)
Using journalctl instead of hunting for the appropriate log files in /var/log seems to offer some degree of platform unity. Maybe it's not the right answer, but it works well enough. My primary complaint is how long it can take on a noisy system to filter a large log, but it's not exceptionally noisome. And most of the time, I just want to follow the log.
> I would agree with an approach to have systemd send log events to e.g. journald (or syslog-ng, or whatever the user wanted) to separate that functionality out, however.
While it probably isn't what you mean as you attach the caveat of task separation, it is possible for syslog-ng to collected logging from systemd's journal. It's a little flaky but it works. :)
>I do suspect binary logging isn't the only reason hkhanna dumped Arch for Debian. It smells of nitpickery to me
You are right, of course. Binary logging was not the only reason I ultimately switched. The big reason was that I needed a distribution that did not change so frequently (although I love what Arch is doing). Stability was so important for my use case, though, that I ended up switching to Debian Stable.
Although journalctl does work well, I admit, I wonder if the advantages could have been accomplished without the binary file format. So many people have homegrown log parsing tools that need to be re-written to handle a binary file format. Or at least re-written to call journalctl as an intermediary.
You are right, of course. Binary logging was not the only reason I ultimately switched. The big reason was that I needed a distribution that did not change so frequently (although I love what Arch is doing). Stability was so important for my use case, though, that I ended up switching to Debian Stable.
Although journalctl does work well, I admit, I wonder if the advantages could have been accomplished without the binary file format. So many people have homegrown log parsing tools that need to be re-written to handle a binary file format. Or at least re-written to call journalctl as an intermediary.
> The big reason was that I needed a distribution that did not change so frequently
Yeah, this is a perfectly valid reason and also why I tend not to use Arch on servers (as much as I would like to otherwise, though...). Both have their use cases, and I love Arch for desktops, but it's not a particularly great match for situations where rapid, near-constant change is unwarranted, unnecessary, or problematic. Arch is also a distro that requires a fair amount of love and care--failure to do so, as a friend of mine learned last year when I had to help him update a year old install, can result in an upgrade path that's nearly impossible. Fortunately, ARM was still up at the time.
There certainly is something to be said for stable distros with a strict upgrade path.
> I wonder if the advantages could have been accomplished without the binary file format.
I don't know. I'll be honest: I can't think of a time when I've actually made use of the filtering features of journalctl and systemd's binary format. Nearly all circumstances that have required me to examine the log are usually done by observing the last few entries or manual searches. To that extent, I'd agree that it bears a certain sense of overkill.
That said, it is still possible to feed syslog-ng entries from systemd. So I can't think of a reason to fret much over specific tooling or the likes that expects text logs.
IMO, the udev/systemd relationship, repo, etc. is probably a much more valid complaint, especially for distributions that aren't planning on using systemd in the near future as it likely has some impact on developer/maintainer workload. Although from a sysadmin perspective, the benefits of systemd are very real and material, at least in my experience.
Service introspection could probably use some work to simplify, so that's another complaint. I can never remember the commands half the time. ;)
Yeah, this is a perfectly valid reason and also why I tend not to use Arch on servers (as much as I would like to otherwise, though...). Both have their use cases, and I love Arch for desktops, but it's not a particularly great match for situations where rapid, near-constant change is unwarranted, unnecessary, or problematic. Arch is also a distro that requires a fair amount of love and care--failure to do so, as a friend of mine learned last year when I had to help him update a year old install, can result in an upgrade path that's nearly impossible. Fortunately, ARM was still up at the time.
There certainly is something to be said for stable distros with a strict upgrade path.
> I wonder if the advantages could have been accomplished without the binary file format.
I don't know. I'll be honest: I can't think of a time when I've actually made use of the filtering features of journalctl and systemd's binary format. Nearly all circumstances that have required me to examine the log are usually done by observing the last few entries or manual searches. To that extent, I'd agree that it bears a certain sense of overkill.
That said, it is still possible to feed syslog-ng entries from systemd. So I can't think of a reason to fret much over specific tooling or the likes that expects text logs.
IMO, the udev/systemd relationship, repo, etc. is probably a much more valid complaint, especially for distributions that aren't planning on using systemd in the near future as it likely has some impact on developer/maintainer workload. Although from a sysadmin perspective, the benefits of systemd are very real and material, at least in my experience.
Service introspection could probably use some work to simplify, so that's another complaint. I can never remember the commands half the time. ;)
That sounds handy, but I wonder if json could provide the same thing.
It is binary so you can quickly filter (indexes). E.g. on service, but any field is possible. You can output it in json. systemctl gives you the last lines of service output exactly because of the indexes + binary file. Using json for that just because "OMG binary" seems strange. Json is not readable.
[deleted]
Postgres can query inside json. To be honest I don't care that much about logging as long as it is saved to disk.
I am not a fan of binary logs, either.
I am a fan of more consistenly structured/filterable logs that are easier to pin to their service. I'm an even bigger fan of the fact systemd goes to a more Unixy model of not making every daemon author re-invent the logging wheel for their double-forking daemon.
I am a fan of more consistenly structured/filterable logs that are easier to pin to their service. I'm an even bigger fan of the fact systemd goes to a more Unixy model of not making every daemon author re-invent the logging wheel for their double-forking daemon.
Bull, we've had a syslog facility for ages.
Which would be why pretty much ever web server and mail server choose to implement their own logging mechanism, right?
Or perhaps its that having to bundle your logs together into a comparative handful of categories and then sort through them is less than entirely optimal.
Or perhaps its that having to bundle your logs together into a comparative handful of categories and then sort through them is less than entirely optimal.
Daemons implementing their own logging in addition to syslog is a feature, not a bug. With journald, even more so.
My understanding is that for a number of services (postgresql comes to mind) the issue was that syslog isn't implemented on all platforms, notably Windows.
I too was put off at first when Arch moved from rc.conf to systemd as I found rc.conf simpler to use, though I trust there was a good reason to move as they are pretty good at managing the distribution.
Rachel mentioned why binary logging makes sense for complicated logs that require rather tricky state machines/regex. Apache for example, in her case.
http://rachelbythebay.com/w/2012/02/10/logs/
Rachel mentioned why binary logging makes sense for complicated logs that require rather tricky state machines/regex. Apache for example, in her case.
http://rachelbythebay.com/w/2012/02/10/logs/
Systemd is a decent init system, fast (ssd+systemd=10 sec from button press), and dependency resolution is quite good. The units are configurable in the /etc/systemd directory and take precedence over the (/usr)/lib/systemd units. It is not perfect but it works. I believe it is somewhat better that sysvinit especially in dependencies. You just need an After=foo.service or Wants=foo.service and systemd takes care of everything. I know this is supported by sysvinit but the whole numbering thing always seemed like a hack to me. Systemd gets it right.
That being said, I can't say I am a fan of the integration of other services within systemd, it seems somewhat bloated. And I certainly don't like the whole journal thing. When the journal becomes bigger it becomes considerably slower. And I don't get the point for binary logs. I mean, logs are just text, why overcomplicate things?
I've been using it for over a year in Arch it is really good as an init daemon, I believe it is definitely a step to the right direction especially for desktops/workstations, probably for mobile as well. The rest of the package though, I am not so sure.
That being said, I can't say I am a fan of the integration of other services within systemd, it seems somewhat bloated. And I certainly don't like the whole journal thing. When the journal becomes bigger it becomes considerably slower. And I don't get the point for binary logs. I mean, logs are just text, why overcomplicate things?
I've been using it for over a year in Arch it is really good as an init daemon, I believe it is definitely a step to the right direction especially for desktops/workstations, probably for mobile as well. The rest of the package though, I am not so sure.
It's quite trivial to override services - you just cp /usr/lib/systemd/system/myservice to /etc/systemd/system/myservice and edit it there.
This has the advantage of not having you screw up the actual package managed files, too.
This has the advantage of not having you screw up the actual package managed files, too.
The "systemd" project really isn't about having just an init system. systemd by itself is an init, but the project includes several other tools related to managing the system as a whole beyond just starting services. The project integrates a bunch of utilities that each accomplish something on their own.
To me, it makes perfect sense for these tools to be tightly integrated, both with each other and with the OS kernel, simply because they are interrelated and integration leads to greater stability and a less fragile system overall.
No doubt upstream could be less hostile towards attempts to distill out the portable bits, but I can understand their viewpoint... Systemd is ultimately about making Linux-based systems run well.
To me, it makes perfect sense for these tools to be tightly integrated, both with each other and with the OS kernel, simply because they are interrelated and integration leads to greater stability and a less fragile system overall.
No doubt upstream could be less hostile towards attempts to distill out the portable bits, but I can understand their viewpoint... Systemd is ultimately about making Linux-based systems run well.
> simply because they are interrelated and integration leads to greater stability and a less fragile system overall.
This is completely counter to the way we actually design software - which is to create modular, loosely coupled components, using abstractions which allow them to adapt or be replaced, and trying to make as few assumptions as we can, since we can't predict the future. It's well understood that tight coupling is what leads to fragility and maintenance nightmares, due to decades of experience writing and maintaining such abominations.
This is completely counter to the way we actually design software - which is to create modular, loosely coupled components, using abstractions which allow them to adapt or be replaced, and trying to make as few assumptions as we can, since we can't predict the future. It's well understood that tight coupling is what leads to fragility and maintenance nightmares, due to decades of experience writing and maintaining such abominations.
That just sounds like an argument from idealism. Writing software is all about tradeoffs, and not supporting non-Linux systems is one tradeoff that systemd makes, for a good reason.
In this case, close integration with the kernel is required to be able to produce the features envisioned.
You might perhaps find some instances where systemd components are needlessly interrelated, but the claim that it isn't modular is mostly a myth.
In this case, close integration with the kernel is required to be able to produce the features envisioned.
You might perhaps find some instances where systemd components are needlessly interrelated, but the claim that it isn't modular is mostly a myth.
If Lennart Poettering were building a bicycle, it would have multiple seats, 4 wheels, engine, conditioner, bar, sat tv, console and small pool on it. You would be able to remove most of it tho... maybe. If you try to remove the engine, it would complain that modern all bicycles are broken without engines, but you can do it at your own risk.
Your comparison makes no sense at all. E.g. systemd dependencies are minimal.
My take: if he builds a bicycle it would be lean, fast and he'd analyse all existing designs to make that happen.
My take: if he builds a bicycle it would be lean, fast and he'd analyse all existing designs to make that happen.
What do dependencies have to do with anything? If I make a huge massive bloated mess of shit, it isn't suddenly "lean" because I included everything in it rather than linking to it as a dependency.
I suggest investigating systemd then. It isn't bloated. See for instance http://people.debian.org/~stapelberg/2013/06/09/systemd-bloa.... There are a lot of different parts to it.
Note the time of the blog. Saying systemd is bloated is IMO so 2013 :P
Note the time of the blog. Saying systemd is bloated is IMO so 2013 :P
It is bloated. Saying "I am cool with how bloated it is" does not change that.
[deleted]
> not to have everything symlinked to /usr.
an init system should be able to run !BEFORE! /usr is even mounted. Thats the division between /directory and /usr/directory, that /usr is mounted later.
an init system should be able to run !BEFORE! /usr is even mounted. Thats the division between /directory and /usr/directory, that /usr is mounted later.
While that's the theory, it hasn't actually worked in practice on most distros for many years -- rather than having "core stuff in root, extra stuff in /usr", we now have "core stuff in initrd, some extra stuff in root, different extra stuff in /usr, with different distros splitting the latter two up differently", the usr merge actually brings us back to the original clean design of having one "core" place and one "extras" place
And udev, which was folded into systemd.
Does anyone else feel like the only reason to choose systemd is the systemd-journal? That thing is just amazing. The amount of leverage provided by not just appending to a log file somewhere will really get the juices moving in anyone who ever had to write some ad-hoc parser to do the same thing poorly.
If anyone hasn't tried out systemd or journalctl yet, I suggest running a copy of Arch Linux in a virtual machine.
If anyone hasn't tried out systemd or journalctl yet, I suggest running a copy of Arch Linux in a virtual machine.
I think the unit files are another reason. Over the years, my patience with differing ideas of how an initscript should function has waned to the point that it's almost an exercise in frustration if it's not something you regularly write on a platform you seldom use. Unit files alone make systemd a breath of fresh air, because familiarizing yourself with them requires reading only the manpages (and even then only the applicable portions). Even somewhat esoteric behaviors are readily supported without much effort.
I realize initscripts are "simple" in the sense that they borrow immediately from the shell and therefore possess all the advantages (and disadvantages) implied by such. Unfortunately, the notion of a sane initscript seems to be greatly varied in the sense that it depends largely on the distribution and the package maintainers. It isn't that they're necessarily difficult to write as much as having the power of the shell available inclines some authors to do very stupid things.
Comparatively speaking, a unit file should be exceptionally easy to understand at first blush.
I realize initscripts are "simple" in the sense that they borrow immediately from the shell and therefore possess all the advantages (and disadvantages) implied by such. Unfortunately, the notion of a sane initscript seems to be greatly varied in the sense that it depends largely on the distribution and the package maintainers. It isn't that they're necessarily difficult to write as much as having the power of the shell available inclines some authors to do very stupid things.
Comparatively speaking, a unit file should be exceptionally easy to understand at first blush.
I personally prefer systemd to upstart. I just hope that distros (in this case Debian) stick to a technical review and don't get drawn into ad hominem arguments.
For many people it seems the answer to the question posed in the post title is "I would expect it to not be designed/authored by the man behind pulseaudio".
For many people it seems the answer to the question posed in the post title is "I would expect it to not be designed/authored by the man behind pulseaudio".
One argument that is on the edge: Lennart often takes controversial decisions, and in the case of systemd, he refuses portability patches that would allow systemd to run on other unixes than Linux (BSD, Hurd, etc).
Since debian is supposed to be universal (hence, support alternative Unix kernels), choosing systemd barely means that they will have to drop this trait of the distribution.
Since debian is supposed to be universal (hence, support alternative Unix kernels), choosing systemd barely means that they will have to drop this trait of the distribution.
Various features that systemd exposes have no equivalent in other kernels. This makes it impossible. He also explained various times why he refuses non-complete portability patches. This because you get more ifdefs and those are difficult to test/guarantee.
He and Kay claimed that kdbus couldn't be used for Binder as well. Greg thought it was possible. Only after writing kdbus he came to the conclusion that Lennart+Kay had it right.
Suggest to write a complete patch to make systemd work on FreeBSD. I assume you'll discover Lennart is right :-P
He and Kay claimed that kdbus couldn't be used for Binder as well. Greg thought it was possible. Only after writing kdbus he came to the conclusion that Lennart+Kay had it right.
Suggest to write a complete patch to make systemd work on FreeBSD. I assume you'll discover Lennart is right :-P
Actually, feasible or not, I understand and somewhat agree with the fact that they don't want portability patches, it would probably force systemd to be limited to the lowest common denominator and add a lot of complexity to the project.
However, I was wondering if platform-specific forks are a sustainable solution.
However, I was wondering if platform-specific forks are a sustainable solution.
You do not get portable code with ifdefs, you only get 2 non portable programs in the same file.
Upstart also has the issue of being non-portable as of right now, so either way a porting effort has to be actually undertaken (even if Upstart is saying that it's planned)--And Debian is adopting large amounts of systemd for other systems.
Lennart hate is fine, but let systemd stand on its technical merits (or lack thereof, if you really feel that way).
Lennart hate is fine, but let systemd stand on its technical merits (or lack thereof, if you really feel that way).
A lot less non-portable though. There are people that have managed to boot it on kFreebsd.
https://lists.ubuntu.com/archives/upstart-devel/2014-January...
But yes, it is still a work in progress.
https://lists.ubuntu.com/archives/upstart-devel/2014-January...
But yes, it is still a work in progress.
There is an observed situation in evolutionary ecology which the response to Lennart rather reminds me of: the case of the overwhelming singular adaptive advantage. An ecological situation arises which gives one subspecies a huge, but temporary, advantage. So huge that all the other subspecies are competed to extinction. But that advantage was an advantage in only one aspect of survival and now there's a monoculture (which we all know is more fragile against future threats). If there's a latent, (or perhaps overt?), Lennart-phobia, it's that he possesses that one temporary huge advantage which will lead us to a monoculture? ...or not, this is just a curious analogy [shrug]
Up until this debate, I don't think most people were aware that there was even an init-culture to fear turning into a monoculture :)
I think your observation is true though.
I think your observation is true though.
I suspect that if someone actually did the work to port systemd, upstream would be nicer.
I have mixed feelings about the BSD & Hurd ports. "You shall not crucify Debian Linux upon a cross of Hurd" and all that.
I have mixed feelings about the BSD & Hurd ports. "You shall not crucify Debian Linux upon a cross of Hurd" and all that.
I'm pretty much done with Debian because of this. I have one last server to migrate and then I'll never have to deal with the personalities and preaching again.
> I would expect it to not be designed/authored by the man behind pulseaudio
Having used a bunch of Lennart's software, I'm generally coming to the conclusion that he's better at ideas than code -- but once his projects take off and the bugs get fixed, the end result is significantly better than what we had before.
(Using pulseaudio as a specific example, per-app volume controls and easily mixing and matching sound sources and playback devices; seamlessly switching from speakers to headphones to HDMI to a sound server on another PC entirely, etc -- features that are too high-level for the kernel and too low-level for applications, and thus can only be brought into existence by a fairly large change in the audio stack)
Having used a bunch of Lennart's software, I'm generally coming to the conclusion that he's better at ideas than code -- but once his projects take off and the bugs get fixed, the end result is significantly better than what we had before.
(Using pulseaudio as a specific example, per-app volume controls and easily mixing and matching sound sources and playback devices; seamlessly switching from speakers to headphones to HDMI to a sound server on another PC entirely, etc -- features that are too high-level for the kernel and too low-level for applications, and thus can only be brought into existence by a fairly large change in the audio stack)
I'm tired reading about this. It seems to boil down to politics, mostly and features less... The team with the better players will win this.
What is interesting though, is to see how the BSD ecosystem will handle this. Will they adapt in time or they will lag even further behind this?
* NetBSD might reached the point of virtual insignificance long time ago.
* OpenBSD is out of funds - which is a shame considering the software they produced (PF, OpenSSH, OpenNTPD), but their indifference for desktop makes them obsolete.
* FreeBSD (I use it in both my servers) is the most up-to-date but now with system-d/upstart not supported, I think they will run into trouble relative soon.
Darwin said it best: the ones most adaptable to change survive. Of course, the other happy scenario is that given the fact that these OSes power some big-corp servers, these big-corps will step in and fund these projects, I wouldn't hold my breath though.
What is interesting though, is to see how the BSD ecosystem will handle this. Will they adapt in time or they will lag even further behind this?
* NetBSD might reached the point of virtual insignificance long time ago.
* OpenBSD is out of funds - which is a shame considering the software they produced (PF, OpenSSH, OpenNTPD), but their indifference for desktop makes them obsolete.
* FreeBSD (I use it in both my servers) is the most up-to-date but now with system-d/upstart not supported, I think they will run into trouble relative soon.
Darwin said it best: the ones most adaptable to change survive. Of course, the other happy scenario is that given the fact that these OSes power some big-corp servers, these big-corps will step in and fund these projects, I wouldn't hold my breath though.
I'm pretty sure that FreeBSD are ressurecting their OS X-derived launchd experiment to get a better init.
The argument that sysv "is Unix" is pretty much horseshit at this point. Solaris and OS X don't use a sysv style init, most of Linux doesn't any more (either upstart or systemd). At this point one is redefining "real Unix" down to NetBSD and HP-UX.
The argument that sysv "is Unix" is pretty much horseshit at this point. Solaris and OS X don't use a sysv style init, most of Linux doesn't any more (either upstart or systemd). At this point one is redefining "real Unix" down to NetBSD and HP-UX.
Here's the link to the port-in-progress of launchd to FreeBSD, if anyone is curious:
https://github.com/rtyler/openlaunchd
https://github.com/rtyler/openlaunchd
The sane init is one of the main selling points of the *BSDs for me. I don't want any of this make-my-desktop-linux-boot-two-seconds-faster bullshit on my servers.
You seem to be under the mistaken impression that netbsd and openbsd users demand the lastest in temporary, throw away, bloated, buggy shitware that linux distos are jumping on at any given moment. The fact that they don't use that crap is their main selling point. And no, netbsd is not "virtual insignifance", openbsd is not out of funds (they got over $100k just when they asked for $20k recently, and idiots take that to mean they are dead?). FreeBSD is not the most up to date, it is the most linux-like. Why do you think it is so impossible for them to also start using this pile of shit if that is what their userbase wants?
> netbsd is not "virtual insignifance"
Who uses NetBSD?
Who uses NetBSD?
I would prefer a parallelized dependency-based init system over an event-triggered system. I want to be able to say "start openssh" and have networking start with it automatically, for example, like Gentoo used to have (or still does?).
I would prefer to do my own thinking, not have the computer try to outsmart me. It can't, and it always hurts my sleepcycles when it tries.
You already have an init system that does some level of "thinking" for you. You had to learn how that one works, you just don't want to learn a new one.
> You had to learn how that one works,
> you just don't want to learn a new one.
That conclusion was quickly drawn. :)
That conclusion was quickly drawn. :)
I know! I still can't stand that my computer just boots when I turn it on. I liked it when I had to input my boot code in binary with the switches on the front.
Also, I refuse to use "make" because I prefer typing long and complicated gcc lines and working out all the header file dependencies in my head. Why on earth would I trust my computer to do that for me?
Also, I refuse to use "make" because I prefer typing long and complicated gcc lines and working out all the header file dependencies in my head. Why on earth would I trust my computer to do that for me?
I'll try to explain my point of view. Without resorting to sarcasm.
My experience is with kudzu, magically finding new networkcards all of a sudden. Or udev, which then automagically turns these new found cards into eth12, instead of eth0.
Too often I have 'smart' systems trying to figure out what should be done, and fail miserably. Especially when trying to debug a software-problem, in an enterprise production landscape.
No, I do NOT want systemd to 'figure out for me' that I need network when I start sshd. Because sometimes I don't NEED a running network. When debugging a broken server, I don't want to have to first dive into the configurations of systemd trying to figure out all the dependencies, and stopping systemd from starting, or stopping them.
My experience is with kudzu, magically finding new networkcards all of a sudden. Or udev, which then automagically turns these new found cards into eth12, instead of eth0.
Too often I have 'smart' systems trying to figure out what should be done, and fail miserably. Especially when trying to debug a software-problem, in an enterprise production landscape.
No, I do NOT want systemd to 'figure out for me' that I need network when I start sshd. Because sometimes I don't NEED a running network. When debugging a broken server, I don't want to have to first dive into the configurations of systemd trying to figure out all the dependencies, and stopping systemd from starting, or stopping them.
You can force it if you want to, but in normal operations, starting sshd without waiting for the network to appear is just flat out wrong. It will not work.
So once in a blue moon you want to run systemctl with "--ignore-dependencies"? That's a much different argument than "would prefer to do my own thinking, not have the computer try to outsmart me."
You're right, it's not smarter than you—but it does has a better memory than you. To want things dumb for the 1% case seems stupid to me and an untenable position to take, hence my initial sarcasm.
So once in a blue moon you want to run systemctl with "--ignore-dependencies"? That's a much different argument than "would prefer to do my own thinking, not have the computer try to outsmart me."
You're right, it's not smarter than you—but it does has a better memory than you. To want things dumb for the 1% case seems stupid to me and an untenable position to take, hence my initial sarcasm.
> You can force it if you want to, but in normal operations, starting sshd without waiting for the network to appear is just flat out wrong. It will not work.
There is no such thing as flat out wrong.
Troubleshooting other peoples servers I need to work around many different situations. A '--ignore-dependencies" is nice, but won't cut it as it depends greatly on the situation at hand. Perhaps sshd also has a dependency not just on networking but also on slapd or something. Perhaps I do not want networking, but I do want slapd to be running.
systemd having a better memory than me is, well, true. It doesn't solve my problems for me though. The Windows Registry also has a better memory than me, but I prefer my configs in a textfile I can parse, grep and maintain with any tool I choose. Same with systemd: better memory than me, but I do not really need a better memory than me. My memory is good enough.
But meanwhile the world has made up it's mind and in all likelyhood Debian will be using systemd. So I might as well start the learning curve and be done with it.
There is no such thing as flat out wrong.
Troubleshooting other peoples servers I need to work around many different situations. A '--ignore-dependencies" is nice, but won't cut it as it depends greatly on the situation at hand. Perhaps sshd also has a dependency not just on networking but also on slapd or something. Perhaps I do not want networking, but I do want slapd to be running.
systemd having a better memory than me is, well, true. It doesn't solve my problems for me though. The Windows Registry also has a better memory than me, but I prefer my configs in a textfile I can parse, grep and maintain with any tool I choose. Same with systemd: better memory than me, but I do not really need a better memory than me. My memory is good enough.
But meanwhile the world has made up it's mind and in all likelyhood Debian will be using systemd. So I might as well start the learning curve and be done with it.
Not sure how this applies. I assume you automate some things that you could simply type in by hand every single time, why not this?
Because it murks things up. Same as network-manager pointing /etc/resolv.conf to 127.0.01, and then run it's own resolving dnsmasq on that address. It's trying to be too smart in resolving a problem. And now, when I cannot resolve a hostname, can't even edit /etc/resolv.conf, or trust it's contents. I now have to look into /etc/dnsmasq.conf, interpret it, find out where it stores the real resolv.conf file and have a look in that file.
It's overcomplicating things.
Yes, I love automation. I puppetize and ansible-ize the hell out of my serverparks. But udev, kudzu, network-manager and now systemd: they're solving the problem by creating more complex problems.
I just have very bad experiences with automagic systems trying to outsmart me. And failing miserably, costing me a saturday-evening with the missus. And outsmarting me ain't even that hard!
It's overcomplicating things.
Yes, I love automation. I puppetize and ansible-ize the hell out of my serverparks. But udev, kudzu, network-manager and now systemd: they're solving the problem by creating more complex problems.
I just have very bad experiences with automagic systems trying to outsmart me. And failing miserably, costing me a saturday-evening with the missus. And outsmarting me ain't even that hard!
My smart phone automatically silences from the Bluetooth speakers when I plug in my headphones. When I'm listening to my headphones and remove them, the music automatically pauzes. It thinks for me, and I think it is great!
I can also gives loads of similar examples for just a computer, but above behaviour is something I really appreciate every day.
I can also gives loads of similar examples for just a computer, but above behaviour is something I really appreciate every day.
systemd does exactly that.
Not exactly. It does do stuff concurrently, but not always in "parallel" since it may only have 1 core available or not get scheduled across multiple cores. I disagree with that particular wording of the systemd description because it perpetuates the concurrency == parallelism misunderstanding. Not that I don't like systemd, I've been using it for over a year and haven't had any issues, and its concurrent model does seem to improve boot times somewhat and there's no more figuring out how to order your daemons since it handles that for you.
And there is no need to make it parallel. There is very little computation the init itself needs to do - most of the time it is waiting for things to happen, and the time it takes to react to events is negligible.
Personally, I prefer systemd. I have quite a shallow understanding of upstart, but even so I find a dependency-based model much easier to reason about than an event-based model.
I'm quite sure either would be a massive improvement over sysvinit though.
I'm quite sure either would be a massive improvement over sysvinit though.
PulseAudio was wonky in 2005, therefore sysvinit forever.
PulseAblipudio is still wonky in 2014 if you need to use it alongside JACK.
You can easily pause pulseaudio while the Jack program is running. I saw the development discussion to make that happen automatically. Further, there are various ways to do that. Seems at most a problem with how it is integrated in your distribution.
You don't even need to pause PA. See, for example, what cadence[^1] does to bridge PA and Jack.
[1]: http://kxstudio.sourceforge.net/Applications:Cadence
[1]: http://kxstudio.sourceforge.net/Applications:Cadence
Containers of Debian 7 (openvz) start in seconds, with sysv init. My laptop dito.
What am I missing? Android? That takes forever to boot and I have no idea why. Could be that systemd fixes that problem.
To me, Systemd and upstart come across as modern-for-the-sake-of-being-modern.