Linux syslog on the way out?(itworld.com)
itworld.com
Linux syslog on the way out?
http://www.itworld.com/it-managementstrategy/227291/linux-syslog-may-be-way-out
16 comments
A second read makes me believe it solves no problem except disk space usage. The fact it's binary and signed protects against nothing - whoever has control of the machine also controls the signing keys of the log and can generate whatever log they want, with whatever event they want.
Are we doomed to repeat every stupid mistake Microsoft made? Is the "if you don't know Windows you are doomed to reimplement its bugs" the new thing?
Are we doomed to repeat every stupid mistake Microsoft made? Is the "if you don't know Windows you are doomed to reimplement its bugs" the new thing?
The solution to integrity, IMO, is distributing the log (multiple independent logservers). You can hash individual line items if you want, or sign them, or do any number of other things to make it easier to find discrepancies. There's the problem of log lossieness in general, which is a tough nut to crack. Some form of log entry UUID might also be useful (though hostname + millisecond/jiffy timestamp comes close).
And count me in among those who find the bug-for-bug reimplementation of Microsoft's fatal errors just a tad annoying.
And count me in among those who find the bug-for-bug reimplementation of Microsoft's fatal errors just a tad annoying.
> The solution to integrity, IMO, is distributing the log (multiple independent logservers).
Isn't this already solved by rsyslogd?
Isn't this already solved by rsyslogd?
Yes.
This is a waste of time. Syslog is the standard for logging on *nix. You will bork everything if you replace it with something incompatible. Everybody knows this. So whatever you replace it with must be Syslog compatible. If that's the case ... there are numerous alternatives to choose from. Like rsyslog. Which is already standard on RHEL, nevermind Fedora. Rsyslog has this nifty feature where you can immediately transfer all logs via TCP or UDP to another host. Hacking problem solved. Stop fixing things that aren't broken.
Not to mention that Journal will depend on systemd which is Linux-only :D
I disagree. A lot of these old *nix tools are showing their age and could use a refresh based on known deficiencies. Creating new tools won't harm the old ones so I don't see a problem.
> Creating new tools won't harm the old ones
Except when they become the default tool despite known deficiencies the previous ones didn't have.
Except when they become the default tool despite known deficiencies the previous ones didn't have.
What are a few examples of other "old *nix tools" that are showing their age and have "known deficiencies?"
init / sysvinit / upstart / systemd: dependencies, reliable process monitoring & restarts, parallelization.
cron: take your pick, but I'd like a scheduler which would produce useful email / messaging reports when things go wrong, and shut the fuck up when they don't, while logging activity in markedly more detail than current jobs do. To accomplish this today you'd need to follow a strict template on your actual cronjobs. Stuff like: what host am I on, what crontab am I being run out of, what user am I running as, what script(s) / program(s) am I invoking.
nfs: performance, reliability, security, arbitrary group limits.
Generally: performance monitoring. We're still relying on top, uptime, vmstat, iostat, strace, ltrace, (sometimes systemtap/dtrace), wireshark, netstat, lsof, and related tools. There are a few new tricks coming out, and some ways to integrate data from multiple sources, but often tuning systems (especially where VMs or application engines are concerned) resembles a seance / archaeological dig / bro knowledge / myth / wishful thinking / Republican plank. For the love of Dennis, it's been 40 years, give me some goddamned insights.
There are probably a bunch of others, though I'm trying to stick with basics that you might find in The UNIX Programming Environment or similar -- really foundational basics (or reasonable extensions of same).
The barrier to change is that most of this stuff works well enough most of the time, and too much depends on it. I'm seeing a lot of exploration of alternatives around some core systems though, and suspect we'll see significant changes in the next 5-10 years. Possibly even sooner, though history suggests slower adoption.
cron: take your pick, but I'd like a scheduler which would produce useful email / messaging reports when things go wrong, and shut the fuck up when they don't, while logging activity in markedly more detail than current jobs do. To accomplish this today you'd need to follow a strict template on your actual cronjobs. Stuff like: what host am I on, what crontab am I being run out of, what user am I running as, what script(s) / program(s) am I invoking.
nfs: performance, reliability, security, arbitrary group limits.
Generally: performance monitoring. We're still relying on top, uptime, vmstat, iostat, strace, ltrace, (sometimes systemtap/dtrace), wireshark, netstat, lsof, and related tools. There are a few new tricks coming out, and some ways to integrate data from multiple sources, but often tuning systems (especially where VMs or application engines are concerned) resembles a seance / archaeological dig / bro knowledge / myth / wishful thinking / Republican plank. For the love of Dennis, it's been 40 years, give me some goddamned insights.
There are probably a bunch of others, though I'm trying to stick with basics that you might find in The UNIX Programming Environment or similar -- really foundational basics (or reasonable extensions of same).
The barrier to change is that most of this stuff works well enough most of the time, and too much depends on it. I'm seeing a lot of exploration of alternatives around some core systems though, and suspect we'll see significant changes in the next 5-10 years. Possibly even sooner, though history suggests slower adoption.
Well pretty much everything if you ask the plan9 guys. ;)
Over the years a lot of the Linux system infrastructure has been reworked:
/dev --> udev
SysV init --> systemd/upstart
su/sudo --> policykit (for desktop stuff)
sockets/shmem for IPC --> dbus (IPC for desktop apps)
xorg --> wayland (not quite yet)
I personally think cron and atd need to be updated and combined and possibly integrated with init.
If you're not running a server and instead are using Linux for a desktop or laptop then nearly all these daemons could be reconsidered as well.
Of course, sometimes you might decide that known deficiencies are better than unknown deficiencies or that simplicity and familiarity means the old tools are still preferred. FreeBSD is pretty old school and very functional after all. But there is plenty of room for experimentation.
Over the years a lot of the Linux system infrastructure has been reworked:
/dev --> udev
SysV init --> systemd/upstart
su/sudo --> policykit (for desktop stuff)
sockets/shmem for IPC --> dbus (IPC for desktop apps)
xorg --> wayland (not quite yet)
I personally think cron and atd need to be updated and combined and possibly integrated with init.
If you're not running a server and instead are using Linux for a desktop or laptop then nearly all these daemons could be reconsidered as well.
Of course, sometimes you might decide that known deficiencies are better than unknown deficiencies or that simplicity and familiarity means the old tools are still preferred. FreeBSD is pretty old school and very functional after all. But there is plenty of room for experimentation.
CVS?
I believe this is a bad example. Because CVS is only under maintenance there is no new feature getting added and the last stable release was in 2008. Also anyone that does a bit of research on version control software would realize CVS is only around for legacy purposes and there is enough paths to convert to SVN or a more "newish" DVCS.
CVS uses a text-based format. SVN is an "improved" version that uses a binary format. I think the parallels are quite striking.
(FWIW I don't think that syslog is broken, but then I'm a programmer. Source control is a key tool in my craft, so I'm keenly aware of CVS's limitations. If I were a ststems administrator, perhaps I'd feel the same way about syslog?)
(FWIW I don't think that syslog is broken, but then I'm a programmer. Source control is a key tool in my craft, so I'm keenly aware of CVS's limitations. If I were a ststems administrator, perhaps I'd feel the same way about syslog?)
Over the summer I worked at a company with so little institutional knowledge that an argument broke out about whether to use NTP or not, and what to replace it with (in a webapp, natch) to ensure a good timebase. I was so flustered by this line of discussion that I had trouble participating.
What exactly was the argument against NTP?
They weren't sure it was good enough, or good any more, and they were trying to figure out ways of maintaining a timebase completely within the application, from server to server. These were developers, both salary and contract, in their 30s. As a former sysadmin, now programmer, it was surreal.
It's sad that this is basically a microcosm of the way things typically are in our field.
On a compromised system you can't trust any file stored on the system. Be it text log files or binary log files. The person who compromised the system will just dump the entire binary store instead, which is normally what happens to text log files.
I use many commands (grep, tail, head, etc) to view and parse log files daily. And storing the files will just add complexity to a very simple system. Now I don't disagree that syslog needs some further advancements for access control/management. I may just be not thinking a head and what else could be possible; but I just see this interrupting my normal workflow for debugging issues.
I use many commands (grep, tail, head, etc) to view and parse log files daily. And storing the files will just add complexity to a very simple system. Now I don't disagree that syslog needs some further advancements for access control/management. I may just be not thinking a head and what else could be possible; but I just see this interrupting my normal workflow for debugging issues.
Cryptography, properly applied, allows you to verify file integrity. The real problem becomes one of key management however -- if the Journal proposal solves this issue (and it's not a simple one) then it's possible that the "has the integrity of my system logs been preserved/maintained" question can be answered.
Managing keys properly is highly non-trivial, especially on a system in which you want those keys to be available/accessible automatically at boot. Since the proposal works via signing (hashing) rather than decrypting, PKI could be used, but I still smell issues.
Managing keys properly is highly non-trivial, especially on a system in which you want those keys to be available/accessible automatically at boot. Since the proposal works via signing (hashing) rather than decrypting, PKI could be used, but I still smell issues.
Cryptography allows you to verify that somebody who does not possess your private key did not modify the file. A hacker on the same system as a syslogger has the private key of the syslogger. They can "simply" (for varying values of simply) replay the logs again, only with whatever events they want filtered out or modified, and it will pass the crypto checks just fine. You have to offload your logs somewhere else, at which point the whole encryption in the logs themselves are sort of redundant.
Managing keys in this case isn't just "highly non-trivial", it's flat out impossible.
Managing keys in this case isn't just "highly non-trivial", it's flat out impossible.
Using PKI, this isn't necessarily true. Even constructing a system with simple one-way hashes might be sufficient.
If the logging system is based on a hash of the current and prior records, you get chaining and integrity validation prior to the attack (one problem of attacks is wiping out immediate or all prior history). If you've got remote storage / logging, particularly on multiple systems with independent trust models (compromising system A doesn't necessitate compromise of B and/or C), you're in better shape still.
A signature which required periodic updating of a key from a key provider via a mechanism that prevents later key reuse would mean that past records couldn't be fabricated or modified after the fact. This might require passing data through two or more systems to accomplish the signing in a trusted fashion (not particularly amenable to high-rate logging).
The verification key need not be on the logging system at all (and ideally wouldn't be). You'd access and verify logs from a standalone hardened and highly trusted system (say: booted from known good static media).
Much of which is good for the extreme case, but as I note in my other comments, isn't particularly practical for day-to-day needs. Which means it's also likely to be unfamiliar to technical staff, rarely practiced, rarely exercised, and buggy.
If the logging system is based on a hash of the current and prior records, you get chaining and integrity validation prior to the attack (one problem of attacks is wiping out immediate or all prior history). If you've got remote storage / logging, particularly on multiple systems with independent trust models (compromising system A doesn't necessitate compromise of B and/or C), you're in better shape still.
A signature which required periodic updating of a key from a key provider via a mechanism that prevents later key reuse would mean that past records couldn't be fabricated or modified after the fact. This might require passing data through two or more systems to accomplish the signing in a trusted fashion (not particularly amenable to high-rate logging).
The verification key need not be on the logging system at all (and ideally wouldn't be). You'd access and verify logs from a standalone hardened and highly trusted system (say: booted from known good static media).
Much of which is good for the extreme case, but as I note in my other comments, isn't particularly practical for day-to-day needs. Which means it's also likely to be unfamiliar to technical staff, rarely practiced, rarely exercised, and buggy.
If they are on the box and you don't have your logs offloaded, they can at the very least destroy the logs, and barring your complicated key updating system which would itself require lots of careful review, probably forge anything they want. But destruction is a pretty useful capability itself.
If the logs are being offloaded off the system, then you don't need any of this either because they are already being put somewhere the hacker can't touch. (Of course if they hack that system then you're in real trouble, but that's just moving the problem around; ultimately if your hacker owns everything, you've really, really lost.)
There's no inbetween state where all this complexity solves any problem. Either you've got off-box storage and you don't need it, or you don't and you've already lost if a hacker gets root.
If the logs are being offloaded off the system, then you don't need any of this either because they are already being put somewhere the hacker can't touch. (Of course if they hack that system then you're in real trouble, but that's just moving the problem around; ultimately if your hacker owns everything, you've really, really lost.)
There's no inbetween state where all this complexity solves any problem. Either you've got off-box storage and you don't need it, or you don't and you've already lost if a hacker gets root.
I agree with your points here.
I don't see the proposed journal solution really solving on-system log integrity.
There are a few other points (processes impersonating other processes, e.g., or extended logging formats) which might be better supported in something other than a traditional syslog. But I'm absolutely not sold on Journal.
I don't see the proposed journal solution really solving on-system log integrity.
There are a few other points (processes impersonating other processes, e.g., or extended logging formats) which might be better supported in something other than a traditional syslog. But I'm absolutely not sold on Journal.
I'd love to see something like that implemented - for the benefit of having semi-structured messages... Oh wait, there are lots of daemons which already do that. And they use properly documented formats, like GELF. And they support forwarding over network. And they store data inside a proper database, instead of yet-another-file-format.
I'd much rather they simply blessed one of the existing solutions (greylog2, fluent, loads of others) and created a compatible minimal daemon which can be included by default in a distribution. Currently they're neither providing new good solution, nor solving the structured log problem (flat k/v? why not just go json/bson?). They're also trying to solve single-host security, which seems pretty close to impossible without moving logging into something like TPM - just send the logs outside, so that the logging channel is append only.
There is also one point which makes me really worried about what the hell are they thinking... "The timestamps generally do not carry timezone information, even though some newer specifications define support for it." Simple solution is to log in UTC, always, everywhere. Run your servers in UTC too - unless it's your local desktop, there's no reason not to.
I'd much rather they simply blessed one of the existing solutions (greylog2, fluent, loads of others) and created a compatible minimal daemon which can be included by default in a distribution. Currently they're neither providing new good solution, nor solving the structured log problem (flat k/v? why not just go json/bson?). They're also trying to solve single-host security, which seems pretty close to impossible without moving logging into something like TPM - just send the logs outside, so that the logging channel is append only.
There is also one point which makes me really worried about what the hell are they thinking... "The timestamps generally do not carry timezone information, even though some newer specifications define support for it." Simple solution is to log in UTC, always, everywhere. Run your servers in UTC too - unless it's your local desktop, there's no reason not to.
Plan 9 has an extra permission for files; "append." When set, this means programs are only allowed to append text to the end of the file. Both simplifying logging (just open the file and write) and making it more secure (no arbitrary writes.)
I wonder if Lennart and Kay would accept a solution that simple...
I wonder if Lennart and Kay would accept a solution that simple...
That's the curious thing about syslogging to a network log server, it is append only (assuming the log server isn't compromised.)
It is an existing solution that simple.
It is an existing solution that simple.
Do these guys even have an editor? From the article:
...nature of the syslog, which basically excepts text strings in whatever form the application...
...one daemon may send information about an even in one way, and another daemon in a completely different way...
...nature of the syslog, which basically excepts text strings in whatever form the application...
...one daemon may send information about an even in one way, and another daemon in a completely different way...
Just because you make something "binary" does not mean it is inherently secure. As pointed out elsewhere, syslogd is a pretty reliable, simple and flexible foundational piece of infrastructure. Changing this with something un-tested and not entirely understood would, imho, be a Very Bad Thing(TM).
It makes me smile sadly when one of the problems cited is "Syslog is only one of many logging systems on a Linux machine", and the solution proposed is the addition of another logging system.
I think they're proposing to remove all the others.
There is no method currently defined to forward logs to another machine, so the journal is definitely prone to a local attack wiping it out (you typically have to be root to wipe log files anyway). The only method mentioned to move logs off-system is scp or rsync or some other method of copying a file over the network, which, to me, is a non-starter. I keep logs both locally and forward them to a trusted host, just in case.
I'm neutral on the binary format of the log; I can see both sides of the argument.
I'm neutral on the binary format of the log; I can see both sides of the argument.
Syslog is definitely not on its way out. It's been in use for 30 years because it works. Perhaps this is rather Redhat trying its hand at the embrace & extend game by introducing proprietary tools? Forcing staff to train on such technologies (with pretty yet dysfunctional guis) will make migrating to other systems harder.
As for the free-form text "problem" this solution claims to address: Different daemons produce logs in different format simply because one's data won't fit in another's schema. And you can actually live with postfix and qmail producing logs in different formats about the same event because you have strictly one of them installed in your server. And in my experience, switching from one to another is a huge undertaking where rewriting your regexp hacks is a drop in a bucket.
As for the free-form text "problem" this solution claims to address: Different daemons produce logs in different format simply because one's data won't fit in another's schema. And you can actually live with postfix and qmail producing logs in different formats about the same event because you have strictly one of them installed in your server. And in my experience, switching from one to another is a huge undertaking where rewriting your regexp hacks is a drop in a bucket.
Mm. So many nails in this coffin.
Many times I've had a need to grep or tail a log in a hurry. This kind of change compromises the entire modus operandi of good Unix admins.
There must be tens of thousands of Unix-based tools out there that use the syslog(3) interface and I'm damn sure that most of them will not change Just Because Someone At Fedora wants them to.
I don't believe a binary format is any more inherently secure than a textual one. Want non-repudiation of logs? Sign them and send them off-box.
They would be much smarter to introduce incremental improvements i.e. extend the existing syslog protocol to accept a structured data format; introduce a log signing journal for integrity checks.
Many times I've had a need to grep or tail a log in a hurry. This kind of change compromises the entire modus operandi of good Unix admins.
There must be tens of thousands of Unix-based tools out there that use the syslog(3) interface and I'm damn sure that most of them will not change Just Because Someone At Fedora wants them to.
I don't believe a binary format is any more inherently secure than a textual one. Want non-repudiation of logs? Sign them and send them off-box.
They would be much smarter to introduce incremental improvements i.e. extend the existing syslog protocol to accept a structured data format; introduce a log signing journal for integrity checks.
The bit I really don't like is that it enforces systemd - and it sounds like it's part of it. I think one of the strengths of Unix/Linux is the architecture of many simple tools, each doing one job well, and one job only - which this sounds like it violates? Ick.
They're going to need really good import and export tools. If they can provide those, then most of the objections to the change go away. If they can't, then I doubt it will gain traction.
This is the same sort of thinking that is trying to add a registry to linux as a replacement for /etc
If this means that syslog-ng is in the way in, great!
Immediately prior to launching myself into Linux, I briefly toyed with Windows NT 4.0 WS, and among the bigger disappointments of it was the system logging facility and its user interfaces. While *nix logging has its bugaboos, the ability to chain simple tools together to perform tasks (example: grep, cut, and uniq -c on a postfix logfile to give a per-hour summary of mail delivery status by various classifications) provides tremendous flexibility. As compared with the single GUI interface Microsoft provided. I'd hope that Poettering et al won't repeat this bone-headed mistake, but the simple observation that their sample log event lacks ... a date stamp??!!! ... doesn't inspire me.
If there's a useful element from this it's the list of horribles against current syslog weaknesses. Many of these could be addressed by amending syslog and/or providing a bolt-on solution. I don't see a need for a total replacement.
As to the observation that there are currently several (incompatible) logging systems on Linux systems, ob xkcd: http://xkcd.com/927/