Ancient Linux servers: The blighted slum houses of the Internet(arstechnica.com)
arstechnica.com
Ancient Linux servers: The blighted slum houses of the Internet
http://arstechnica.com/security/2014/03/ancient-linux-servers-the-blighted-slum-houses-of-the-internet/
7 comments
A browser add-in that would check and warn/block loading when the OS and language versions aren't sufficiently patched might be useful security mechanism.
Would it be feasible to write a virus that only exploits vulnerabilities more than, say, 18 months old, to either encourage or force sysadmins to upgrade?
If you get access, then you can look for ssh keys and whatnot to figure out who to send form letters to. "Your server is vulnerable to x, y, and z: please upgrade a, b, and c." It would be more fun if the virus tried to upgrade a, b, and c itself, though. Even if it fails, a broken server that used to process credit cards is better than a vulnerable one, right?
If you get access, then you can look for ssh keys and whatnot to figure out who to send form letters to. "Your server is vulnerable to x, y, and z: please upgrade a, b, and c." It would be more fun if the virus tried to upgrade a, b, and c itself, though. Even if it fails, a broken server that used to process credit cards is better than a vulnerable one, right?
As we know from recent history, many companies have pursued criminal charges against people who have broken into their systems, even if they had no intention to do harm. And DAs are very happy to prosecute these cases. So this would be a great virus to write if you wanted to add a prison sentence to your resume. (And you could be held responsible for the cost of restoring the servers after your failed upgrades, which could run into the millions of dollars if you damaged enough servers.)
Back in 1996, I heard two different people complain that since Microsoft convinced anti-virus companies to support Windows 95, and drop support for Windows 3.11, new viruses for Windows 3.11 were forcing them to upgrade to Windows 95. At least one of these folks was perfectly happy with Windows 3.11, had a huge stash of Windows 3.11 games on CDs and just plain didn't want to upgrade. He saw the whole thing as a charade to force people to shell out for upgrades.
This is playing out again pretty much word-for-word with the termination of support for Windows XP.
It would have been a more valid complaint for someone compelled to upgrade from Win3.1 to Windows 95.
At this stage, Windows XP users are like Japanese soldiers stranded on an island, growing old not knowing they lost the war. Except without the excuse of being on an island.
At this stage, Windows XP users are like Japanese soldiers stranded on an island, growing old not knowing they lost the war. Except without the excuse of being on an island.
I run Win7 on my main machine but I have laptop that still runs XP (and which isn't connected to the internet, in general), because I have a large piece of audio hardware whose driver support stopped with XP but which is otherwise fine, and which I have no desire to replace ahead of time. I'm not alone in this; while XP is far from ideal as an audio production environment, later versions of Windows suffer from really terrible MIDI (control protocol for synthesizers) drivers with unreliable timing.
Of course I could get a Mac, bt I'd really rather not switch platforms or buy a bunch of extra expensive hardware for this one purpose. I could also run Linux, but until very recently the selection of music production software has been poor.
That said, I don't have any complaint with MS about their decision to abandon the platform, I think they've done a good job in supporting it this long. I do wish they'd rethink their driver priorities though, MIDI has been around for over 30 years, and being 7-bit ~32kbps it can't be that hard to get decent realtime performance.
Of course I could get a Mac, bt I'd really rather not switch platforms or buy a bunch of extra expensive hardware for this one purpose. I could also run Linux, but until very recently the selection of music production software has been poor.
That said, I don't have any complaint with MS about their decision to abandon the platform, I think they've done a good job in supporting it this long. I do wish they'd rethink their driver priorities though, MIDI has been around for over 30 years, and being 7-bit ~32kbps it can't be that hard to get decent realtime performance.
I don't know what you're doing but I have rock solid midi timing from Windows 7 via a £5 generic MIDI/USB cable to my Korg Triton Studio. I'm using Renoise as a DAW.
XP was crappy - it hit the disk so often and hung.
XP was crappy - it hit the disk so often and hung.
It's a problem on multi-output USB interfaces for a lot of people. I suspect system configuration issues but I don't think you should have to take a barebones system approach to get decent timing. I mainly sequence in hardware so I haven't gone to great lengths to get to the bottom of it.
DYI http://www.gearslutz.com/board/electronic-music-instruments-... although you'll have to wade through some non-techie voodoo interpretations if you peruse the whole thread. Also, bear in mind that this is a particularly fussy demographic. I sometimes think if you are that obsessed then maybe MIDI isn't the protocol for you, but OTOH there isn't much in the way of OSC hardware on the market :-/ It's notable that there's a bit of a CV renaissance underway though, a lot of people prize timing/resolution over versatility.
DYI http://www.gearslutz.com/board/electronic-music-instruments-... although you'll have to wade through some non-techie voodoo interpretations if you peruse the whole thread. Also, bear in mind that this is a particularly fussy demographic. I sometimes think if you are that obsessed then maybe MIDI isn't the protocol for you, but OTOH there isn't much in the way of OSC hardware on the market :-/ It's notable that there's a bit of a CV renaissance underway though, a lot of people prize timing/resolution over versatility.
Depends how they implemented midi and the timers. If it's win32 MIDI API stuff and Win32 timers the messaging isn't real time so you will get jitter. That's going to be worse on win vista and above due to a number of issues.
If it uses ASIO MIDI back end it'll miss all the problematic message loops.
I only use SYSEX and use the Triton Studio sequencer though as its the most awesome thing ever made by man.
If it uses ASIO MIDI back end it'll miss all the problematic message loops.
I only use SYSEX and use the Triton Studio sequencer though as its the most awesome thing ever made by man.
Is there any reason for such a single-purposed machine to be connected to the internet?
Occasionally I grab a software update or somesuch so I'll connect via wifi rather than bothering with USB transfers. It's not in a commercial environment and there's only a few trusted sites I might be visiting.
As someone stuck supporting a Windows XP server, I know we lost the war. We raised a big white flag and rolled out a welcome mat for our new overlords. Still waiting for someone to show up, though.
(To break the metaphor, our server is hooked up to a PCI-X card which runs a radiation sensor. The drivers for the card will not run under anything after XP. The vendor has discussed possibly considering looking into designing a new card that would might run under Vista, but that's probably five years down the road.)
(To break the metaphor, our server is hooked up to a PCI-X card which runs a radiation sensor. The drivers for the card will not run under anything after XP. The vendor has discussed possibly considering looking into designing a new card that would might run under Vista, but that's probably five years down the road.)
Sounds like you need a new vendor? As an alternative, could you run XP in a minimum VM, which has that particular PCI card's address passed directly to the VM? Then make sure that there is no other way into the XP VM (i.e., don't run any other apps on it, and keep it only on the host-only network).
XP Embedded? Still has support.
Not for long. XPe is already at EOL. Support ends in 2016 and license availability ends in 2017.
Our products using XP Embedded are expected to still be in the field well past 2020. We are already making plans to transition to Windows 8 simply because it will soon be impossible to buy XPe licenses.
Our products using XP Embedded are expected to still be in the field well past 2020. We are already making plans to transition to Windows 8 simply because it will soon be impossible to buy XPe licenses.
> Even if it fails, a broken server that used to process credit cards is better than a vulnerable one, right?
"I saw that you dropped your wallet on the floor, so I threw it into the river so nobody could steal your credit cards."
"I saw that you dropped your wallet on the floor, so I threw it into the river so nobody could steal your credit cards."
If returning the wallet (securing the server) is out of the question, throwing my wallet in the river could easily be preferable to leaving it loose. Of course if someone told me that in meatspace I wouldn't believe them... hm.
You should only intentionally kill their server if you're willing to compensate them for the revenue they lost while they try to figure out what the hell happened.
An alternate analogy would be something like: "I saw you forgot to install a lock on door, so I superglued it shut."
An alternate analogy would be something like: "I saw you forgot to install a lock on door, so I superglued it shut."
You don't fix bad behavior with bad behavior.
This is an interesting problem. Too many people just ignore their machine once it "works" and leave it to the wild, or their attention span drops and ignore it. The painful part is when previously "good" internet sites are now injecting sophisticated malware unbeknownst to the owner.
When I read this I can't help but think about how often I run across people in development or design who don't seem to see difficulty in or value of system administration.
It's pretty trivial to turn up systems that work in the most basic sense, but given enough time or load things start to break down.
The skill isn't in making things work, it's making them work efficiently, securely and automating what will keep them in that state while planning ahead to make it as easy as possible to recover, replace, extend and transition from them in the future.
It's pretty trivial to turn up systems that work in the most basic sense, but given enough time or load things start to break down.
The skill isn't in making things work, it's making them work efficiently, securely and automating what will keep them in that state while planning ahead to make it as easy as possible to recover, replace, extend and transition from them in the future.
Sysadmins create an intangible benefit to companies, I've had a lot of jobs come up that are 'DevOps', by which they mean (incorrectly), that I'm a developer who 'knows how to configure apache'.
Someone remarked today that 'Sysadmin' may return to being a term of endearment as they specialise in unfucking the systems created quickly and scaled organically (much like taking on a DBA after 15 years of having a massively growing dataset). it'll probably mean people are far more specialised.
Someone remarked today that 'Sysadmin' may return to being a term of endearment as they specialise in unfucking the systems created quickly and scaled organically (much like taking on a DBA after 15 years of having a massively growing dataset). it'll probably mean people are far more specialised.
See: mainframe operators.
Are you sure that's what sysadmins themselves do? In practice, the "sysadmin" seems to be "that guy who does all the stuff that we don't know how to automate."
The people who do the automation--if it's repeatable, instead of per-site one-off stuff--are just called developers. E.g., the Docker developers, or the Erlang developers.
The people who do the automation--if it's repeatable, instead of per-site one-off stuff--are just called developers. E.g., the Docker developers, or the Erlang developers.
>"Are you sure that's what sysadmins themselves do?"
By my standard yes, that's what they do. Of course, I expect there are a whole host of common definitions and countless pet ones.
>"In practice, the "sysadmin" seems to be "that guy who does all the stuff that we don't know how to automate."
I would call that person an operator or technician with the amount of manual work and a sort of single-threaded skillset being defining characteristics.
>"The people who do the automation--if it's repeatable, instead of per-site one-off stuff--are just called developers. E.g., the Docker developers, or the Erlang developers."
I can't claim to have met many people who are called or would call themselves developers who have the breadth of knowledge in the layers below the interpreter or compiler to manage or automate across many dissimilar systems.
I imagine this is depends on the environment though, the younger, more homogeneous and closer the company product is to the technology they employ the more likely I'd expect what I would consider developers able to handle those matters.
When a company gets older, more segmented and the business functions further from the technologies I could see the role being more that of a sysadmin.
By my standard yes, that's what they do. Of course, I expect there are a whole host of common definitions and countless pet ones.
>"In practice, the "sysadmin" seems to be "that guy who does all the stuff that we don't know how to automate."
I would call that person an operator or technician with the amount of manual work and a sort of single-threaded skillset being defining characteristics.
>"The people who do the automation--if it's repeatable, instead of per-site one-off stuff--are just called developers. E.g., the Docker developers, or the Erlang developers."
I can't claim to have met many people who are called or would call themselves developers who have the breadth of knowledge in the layers below the interpreter or compiler to manage or automate across many dissimilar systems.
I imagine this is depends on the environment though, the younger, more homogeneous and closer the company product is to the technology they employ the more likely I'd expect what I would consider developers able to handle those matters.
When a company gets older, more segmented and the business functions further from the technologies I could see the role being more that of a sysadmin.
It's difficult to automate: vendor relations, software and package assessment, configuration tuning, troubleshooting, hardware replacement, PCI / SOX / other audit processes, pager response, documentation authoring, procedures development (and refinement), user education (especially on matters affecting security and stability), and what I tend to call technical anthropology: digging into the dim mists of ancient history (anything more than 6 months ago, but in cases stretching back decades) to figure out why a particular design decision was made (often being "hell, seems to work, make it so"), and what might explode spectacularly if it's changed.
But yes, your competent sysadmin is also automating the living fuck-all out of as much as possible. Using standard methods and tools and very clear documentation so as not to create yet more technical debt for the next guy.
And if you're classifying your devs by language, you've got other problems as well.
But yes, your competent sysadmin is also automating the living fuck-all out of as much as possible. Using standard methods and tools and very clear documentation so as not to create yet more technical debt for the next guy.
And if you're classifying your devs by language, you've got other problems as well.
Not by language, no; I meant "Erlang developers" as in "the fellows at Ericsson who make my life easier by repeatably automating distributed failover, at least when you're using their platform." You'd call these fellows, and the Docker ones, and the fellows responsible for writing Chef and Puppet and so on, sysadmins?
I guess I mean more by "automation" than you do, though. Paying some other company to do things like software and package assessment for you, so you just have to consume their customized software distribution and can let the system update whenever it likes, is "automation." Buying standard hardware that you can find pre-tuned configurations for is "automation."
Effectively, anything that moves the company away from maintaining a "system", and closer to being just a tiny core of people who run one tiny thing on top of a large set of frozen, well-known components is "automation."
I guess I mean more by "automation" than you do, though. Paying some other company to do things like software and package assessment for you, so you just have to consume their customized software distribution and can let the system update whenever it likes, is "automation." Buying standard hardware that you can find pre-tuned configurations for is "automation."
Effectively, anything that moves the company away from maintaining a "system", and closer to being just a tiny core of people who run one tiny thing on top of a large set of frozen, well-known components is "automation."
a tiny core of people who run one tiny thing on top of a large set of frozen, well-known components
Your error is in assuming this is possible. Change is a constant. Integrating change with existing systems is the role of the sysadmin.
Your error is in assuming this is possible. Change is a constant. Integrating change with existing systems is the role of the sysadmin.
"that guy who does all the stuff that we don't know how to automate." is the operator.
The sysadmin is the guy who can think and figures out what must be done to keep the system healthy, and decides on how to do that (tooling, manual work, buy hardware or go to the cloud, etc)
The sysadmin is the guy who can think and figures out what must be done to keep the system healthy, and decides on how to do that (tooling, manual work, buy hardware or go to the cloud, etc)
I think the term I more often see used for this role is "system architect" or "systems engineer."
To be more specific:
An architect "lives" in the development staff: they're aware of the internals of the application. Of the system, the architect specifies what must be true of the stack the application runs on--and this can be quite detailed--and might participate in bringing that stack up. But in the system's day-to-day life, the architect need only be aware of APIs for interacting with that stack (which they presumably specified needed to exist.)
An admin, meanwhile, "lives" in the ops staff: they're aware of the internals of the stack. Of the application, they are only given a set of APIs it expects to be made available to it; it's otherwise a black box.
The architect writes scripts, which run alongside the application, to maintain the application's presence on the system. The admin writes scripts, which run directly on the system, to maintain the system within the reference range specified by the architect.
Picture, for example, a de-novo implementation of Search at Google. The Search application has an architect, who specifies what sort of servers Search needs to do its job. The Search architect sends this information off to ops. Ops designates someone "the Search sysadmin", and that person constructs a server farm. They script the farm to correctly cluster tasks, maintain its own health, etc. and train operators in the management of that farm.
In my mind, it's really clear if you think of servers as running something like Docker containers to contain the application: everything outside the container is the sysadmin's job. Everything inside the container is the architect's job.
To be more specific:
An architect "lives" in the development staff: they're aware of the internals of the application. Of the system, the architect specifies what must be true of the stack the application runs on--and this can be quite detailed--and might participate in bringing that stack up. But in the system's day-to-day life, the architect need only be aware of APIs for interacting with that stack (which they presumably specified needed to exist.)
An admin, meanwhile, "lives" in the ops staff: they're aware of the internals of the stack. Of the application, they are only given a set of APIs it expects to be made available to it; it's otherwise a black box.
The architect writes scripts, which run alongside the application, to maintain the application's presence on the system. The admin writes scripts, which run directly on the system, to maintain the system within the reference range specified by the architect.
Picture, for example, a de-novo implementation of Search at Google. The Search application has an architect, who specifies what sort of servers Search needs to do its job. The Search architect sends this information off to ops. Ops designates someone "the Search sysadmin", and that person constructs a server farm. They script the farm to correctly cluster tasks, maintain its own health, etc. and train operators in the management of that farm.
In my mind, it's really clear if you think of servers as running something like Docker containers to contain the application: everything outside the container is the sysadmin's job. Everything inside the container is the architect's job.
I think we mostly agree on this, but your phrasing, to me, makes it look as if you put all the intelligence at the architect side. I think that, ideally, the architect should only specify what is needed, the admin should be in control of the how, and the two should discuss things on level footing.
And that what should be in purely abstract terms. Not "We need an Oracle database", not even "We need a SQL store*, but "We need a transactional store that can do x transactions per second, y a week, stores z TB of data, gets backed up daily, has 75% uptime and can be restored within a week" (note that that goes a lot further than your "only the set of APIs it expects". I guess/expect that you will agree that the non-functional stuff also must be included)
And that what should be in purely abstract terms. Not "We need an Oracle database", not even "We need a SQL store*, but "We need a transactional store that can do x transactions per second, y a week, stores z TB of data, gets backed up daily, has 75% uptime and can be restored within a week" (note that that goes a lot further than your "only the set of APIs it expects". I guess/expect that you will agree that the non-functional stuff also must be included)
Enterprise Linux and version numbering[1]: I'm a bit worried by quotes like
"The mass infection works against servers running version 2.6 of the Linux operating system kernel, some using releases from 2007 or earlier, Lee said."
Some servers running 2.6.x kernels are as secure as anything out there!
[1] https://access.redhat.com/site/security/updates/backporting/
"The mass infection works against servers running version 2.6 of the Linux operating system kernel, some using releases from 2007 or earlier, Lee said."
Some servers running 2.6.x kernels are as secure as anything out there!
[1] https://access.redhat.com/site/security/updates/backporting/
Why is that worrying? I don't read into that an implication that linux 2.6 is insecure, just the terribly vulnerable versions they were using.
The problem is that if you simply say that, something like "2.6.9 is vulnerable", it might not actually be true. Enterprise Linux vendors might actually be running 2.6.9 but with relevant security patches backported.
When it get read out of context by phbs.
This is quite funny because the other day I was asked by a friend to look at the box serving his company site just on a whim. I've never looked at or discussed the tech he was using. Turns out he paid a company for "managed service" in 2004. They were crap so he stopped paying for the managed bit in 2006 but paid the £90/month for the 1U rack space. He was running IMAP and a static site. IMAP stopped working one day so he moved it to hosted Exchange and shrugged it off. The static site was fine and has been updated via FTP since by his design agency who don't know arse from elbow.
So the first thing I did was nmap it. SSH, HTTP, SMTP, IMAP open. Fair enough.
SSH'ed in. Some interesting surprises in there. 10 year old DL360 G2 with one dead disk in the array with CentOS 3 on it. CPU spammed 100% with just about every bit of crap that could work its way through the ancient OpenSSH build (and presumably the cgi-bin and IMAP server (cyrus).
Sooooo... FTP'ed all the files of it, fired up a digital ocean box and locked it down, uploaded the static site to it, changed the A record on his DNS and shut the old box down.
I know this will only postpone the problem for another couple of years but what can you do with people like this? All I got was a "meh" and a crate of lager for my troubles ( granted it was 45 mins work but...)
So the first thing I did was nmap it. SSH, HTTP, SMTP, IMAP open. Fair enough.
SSH'ed in. Some interesting surprises in there. 10 year old DL360 G2 with one dead disk in the array with CentOS 3 on it. CPU spammed 100% with just about every bit of crap that could work its way through the ancient OpenSSH build (and presumably the cgi-bin and IMAP server (cyrus).
Sooooo... FTP'ed all the files of it, fired up a digital ocean box and locked it down, uploaded the static site to it, changed the A record on his DNS and shut the old box down.
I know this will only postpone the problem for another couple of years but what can you do with people like this? All I got was a "meh" and a crate of lager for my troubles ( granted it was 45 mins work but...)
[deleted]
What... what hosting provider will not notice this? Shouldn't their abuse mail address be glowing red?
Do you have any idea of how many abuse addresses are /dev/nulled?
Sadly: black-hat takedown pages are sometimes your best sources of information. A few weeks back, a site I visit periodically was pwned via what appears to have been a Wordpress bug, based on similarities between it and other sites listed by the black-hat hacker. I fired off mail to the standard contact addresses for the site (abuse, webmaster, postmaster), all of which bounced. The listed WHOIS contact actually worked (an argument against domains by proxy), but I also reached out to the upstream hosting provider, who had several other hosts similarly taken down, though other providers were also affected.
Sadly, and from my own experience, monitoring abuse for smaller shops (1-2 man admin team, far too many fires going on) is often a very, very low priority.
Sadly: black-hat takedown pages are sometimes your best sources of information. A few weeks back, a site I visit periodically was pwned via what appears to have been a Wordpress bug, based on similarities between it and other sites listed by the black-hat hacker. I fired off mail to the standard contact addresses for the site (abuse, webmaster, postmaster), all of which bounced. The listed WHOIS contact actually worked (an argument against domains by proxy), but I also reached out to the upstream hosting provider, who had several other hosts similarly taken down, though other providers were also affected.
Sadly, and from my own experience, monitoring abuse for smaller shops (1-2 man admin team, far too many fires going on) is often a very, very low priority.
Shonky company. They wouldn't notice if someone nicked their servers.
Why not host the site on S3?
Because I've had literally piles of trouble with AWS over the years.
Edit: plus I might need to add some dynamic content...
Edit: plus I might need to add some dynamic content...
Dynamic content can be done pretty well with just JavaScript, depending on what your needs are.
As for trouble you've had with AWS, I'd encourage you to take a second look at S3 specifically. It's been pretty bulletproof for a long time now.
As for trouble you've had with AWS, I'd encourage you to take a second look at S3 specifically. It's been pretty bulletproof for a long time now.
That's not dynamic content.
It's not. A company I worked for recently had a 2TiB S3 deployment for archiving insurance documents and the amount of hoops you gave to go through for it to be reliable is incredible (no uploads or downloads are guaranteed and sometimes randomly fail for a while).
It's not. A company I worked for recently had a 2TiB S3 deployment for archiving insurance documents and the amount of hoops you gave to go through for it to be reliable is incredible (no uploads or downloads are guaranteed and sometimes randomly fail for a while).
I am running a box built with Fedora Core 4 (2007 vintage). Never patch any systems. Why would I?
If I am running a service facing the internet, it's custom built and patches would do it no good. Why would I wait for vendor to release a patch? If a service is external, I will watch out for vulnerabilities and rebuild ASAP before any patches are out. Besides, 90% of the time my custom build is not even vulnerable to a particular problem.
If I am NOT running a service, why would I care about patches for it?
Why would I wholesale patch a server anyway? If somebody breaks in and gets a local shell, all is lost anyway. If they are not in, they are dealing with externally facing services only, see above. There are specific and counted number of daemons on every machine.
This whole patch-update thing is misguided and for people that want assurances and no responsibility.
If I am running a service facing the internet, it's custom built and patches would do it no good. Why would I wait for vendor to release a patch? If a service is external, I will watch out for vulnerabilities and rebuild ASAP before any patches are out. Besides, 90% of the time my custom build is not even vulnerable to a particular problem.
If I am NOT running a service, why would I care about patches for it?
Why would I wholesale patch a server anyway? If somebody breaks in and gets a local shell, all is lost anyway. If they are not in, they are dealing with externally facing services only, see above. There are specific and counted number of daemons on every machine.
This whole patch-update thing is misguided and for people that want assurances and no responsibility.
As a security researcher, this approach just confounds me.
I've never had an update break my system, and if someone pushed updates that were broken, I wouldn't trust any old versions of their software any more than the current one.
And we keep finding that people don't update and miss critical vulnerabilities. There may be some admins out there that can independently track and patch every known vulnerability... but that seems like an impossible task for a box with any nontrivial amount of software on it.
And a lot of vulnerabilities aren't widely released. Updates sometimes coincidentally break zero days that were never publicly revealed.
I remember the world where everyone stubbornly refused to leave early versions of IE. Massive problem for security. The Chrome team looked at that and made the call to move to automatic updates. I'm still pretty convinced that's a better world.
You want to run a small box that barely faces the internet where you constantly write your own patches in parallel with the primary software developers, while also researching and patching new vulnerabilities before they are deployed, go for it... but when that becomes the industry norm, I consider it extremely harmful.
Maybe you can pull that off, but most people are not nearly that cool.
I've never had an update break my system, and if someone pushed updates that were broken, I wouldn't trust any old versions of their software any more than the current one.
And we keep finding that people don't update and miss critical vulnerabilities. There may be some admins out there that can independently track and patch every known vulnerability... but that seems like an impossible task for a box with any nontrivial amount of software on it.
And a lot of vulnerabilities aren't widely released. Updates sometimes coincidentally break zero days that were never publicly revealed.
I remember the world where everyone stubbornly refused to leave early versions of IE. Massive problem for security. The Chrome team looked at that and made the call to move to automatic updates. I'm still pretty convinced that's a better world.
You want to run a small box that barely faces the internet where you constantly write your own patches in parallel with the primary software developers, while also researching and patching new vulnerabilities before they are deployed, go for it... but when that becomes the industry norm, I consider it extremely harmful.
Maybe you can pull that off, but most people are not nearly that cool.
I think you misunderstand. It's not that people are pushing out crap updates. Rather, the problem is that when you update one thing on linux you usually end up having to update 100 other things.
I'm in a similar position to the OP, in that I don't generally update linux systems. The problem is that there is no way to simply 'update everything' in linux (at least, not in Centos). yum update certainly doesn't do it - in Centos 5.5 it only gets you php 5.1.x. To get a newer version you have to update it manually or bodge yum.
Then the problem is that many newer packages require a newer glibc or whatever, and that is something that can break your entire system very easily.
I think the root of the problem is that linux isn't very easy to update, unlike Windows.
As long as your linux system is well locked down and you regularly keep an eye on it, I don't see a problem with not updating regularly.
I'm in a similar position to the OP, in that I don't generally update linux systems. The problem is that there is no way to simply 'update everything' in linux (at least, not in Centos). yum update certainly doesn't do it - in Centos 5.5 it only gets you php 5.1.x. To get a newer version you have to update it manually or bodge yum.
Then the problem is that many newer packages require a newer glibc or whatever, and that is something that can break your entire system very easily.
I think the root of the problem is that linux isn't very easy to update, unlike Windows.
As long as your linux system is well locked down and you regularly keep an eye on it, I don't see a problem with not updating regularly.
That makes a lot of sense, thanks. I had always sort of seen Linux as easier to update, since it's a single command, but you're right... that command doesn't necessarily get you all the way. Things are going to vary from distro to distro, and none of them will necessarily roll in the bleeding edge version of whatever thing you want the day it launches. And then, custom code is vital on a lot of machines for a lot of applications, and it will introduce its own dependencies.
That said, these factors really complicate security advice on patch management. If customers could be trusted to lock things down and keep an eye on them, that would be a much better world. And I'm sure a lot of admins out there are more than capable, but I worry about the Dunning Krueger effect catching some admins off guard.
But ultimately, this is just a battle of emphasis more than disagreement. The answer isn't "everyone should always patch everything," it just depends on a lot of factors.
That said, these factors really complicate security advice on patch management. If customers could be trusted to lock things down and keep an eye on them, that would be a much better world. And I'm sure a lot of admins out there are more than capable, but I worry about the Dunning Krueger effect catching some admins off guard.
But ultimately, this is just a battle of emphasis more than disagreement. The answer isn't "everyone should always patch everything," it just depends on a lot of factors.
Everything you said is very general. The article in question talks about "Linux servers" (or that's what I read). Speaking of those, could you explain specifically, what is the point of a wholesale update, as opposed to what is described? What I mean is running specific set of services (which most often built from source anyway) and keeping those services up to date.
What you get in return is both stability and security, since you don't wait for vendor to release a patch and actually understand what the vulnerability is.
With wholesale patching, in fact, you can never be sure whether your system is secure with respect to all published vulnerabilities.
Another interesting detail is, that with services running built from sources often you end up with vulnerabilities not applicable to your configuration.
Oftentimes, you can just tweak the config instead of changing code (and potentially breaking running things).
Software updates is just a cop out for people that are too lazy to pay attention to security.
What you get in return is both stability and security, since you don't wait for vendor to release a patch and actually understand what the vulnerability is.
With wholesale patching, in fact, you can never be sure whether your system is secure with respect to all published vulnerabilities.
Another interesting detail is, that with services running built from sources often you end up with vulnerabilities not applicable to your configuration.
Oftentimes, you can just tweak the config instead of changing code (and potentially breaking running things).
Software updates is just a cop out for people that are too lazy to pay attention to security.
In my experience, the number of admins that say they can just stay on top of the vulnerabilities is greater than the number who actually can or do.
> Everything you said is very general.
Ok, here's specifically how this approach fails in the real world:
A guy is ignoring vulnerabilities that don't seem to apply to his configuration. So there's a kernel flaw that allows privilege escalation. He thought it was no big deal because he doesn't allow a guest login. Then there's a flaw that allows remote users to trigger memory corruption, allowing remote guest access. No risk there, he thought, because guests have no privileges on the box.
You see how the attacker got in?
You might counter that that admin was just too "lazy" to line up all the vulnerabilities and see how they interact. But there were almost 200 vulnerabilities last year just in the kernel. Are you going to conduct the 19,000 security audits required to see how they interact? What about groups of three? What about vulnerabilities in other packages? This workload doesn't just go up linearly.
Also, this approach is weaker against unpublished vulnerabilities. If you're strapping together older software, especially deprecated stuff no one else is using, you're losing one perk of open source software, the many eyes shallow bugs bit. People not using your configuration means no one is going to discover vulnerabilities on it except for attackers. You may think, "good, that makes it harder for attackers," but that's security by obscurity, it doesn't actually hurt attackers very much. Vulnerabilities patched in newer versions of software give insight into vulnerabilities lingering in older versions, so often exploits can be crafted for previous versions far more easily than for newer.
This is why security professionals recommend defense in depth. You don't know which part of your platform is going to break and allow attackers to exploit vulnerabilities that you didn't think were relevant.
Also, for a lot of systems, eventually the admin will change. The guy or gal that follows you will be dependent on the system you set up, and may be less experienced or less capable. If they inherit a patch management system that basically entails, "Become a security expert in addition to your other duties," they are not going to strictly adhere.
You know your system and your situation, so I don't want to sound like your approach can never ever work. But if we're giving advice to the unwashed masses, I think we need more advice tailored for the people you dismiss as "too lazy to pay attention to security," because that describes basically everyone.
PS - When you describe other people's opinions as "lazy cop outs," it can kill discussions. You might watch out for that. HN isn't Reddit, people sometimes bury stuff that has good substance but poor tone.
> Everything you said is very general.
Ok, here's specifically how this approach fails in the real world:
A guy is ignoring vulnerabilities that don't seem to apply to his configuration. So there's a kernel flaw that allows privilege escalation. He thought it was no big deal because he doesn't allow a guest login. Then there's a flaw that allows remote users to trigger memory corruption, allowing remote guest access. No risk there, he thought, because guests have no privileges on the box.
You see how the attacker got in?
You might counter that that admin was just too "lazy" to line up all the vulnerabilities and see how they interact. But there were almost 200 vulnerabilities last year just in the kernel. Are you going to conduct the 19,000 security audits required to see how they interact? What about groups of three? What about vulnerabilities in other packages? This workload doesn't just go up linearly.
Also, this approach is weaker against unpublished vulnerabilities. If you're strapping together older software, especially deprecated stuff no one else is using, you're losing one perk of open source software, the many eyes shallow bugs bit. People not using your configuration means no one is going to discover vulnerabilities on it except for attackers. You may think, "good, that makes it harder for attackers," but that's security by obscurity, it doesn't actually hurt attackers very much. Vulnerabilities patched in newer versions of software give insight into vulnerabilities lingering in older versions, so often exploits can be crafted for previous versions far more easily than for newer.
This is why security professionals recommend defense in depth. You don't know which part of your platform is going to break and allow attackers to exploit vulnerabilities that you didn't think were relevant.
Also, for a lot of systems, eventually the admin will change. The guy or gal that follows you will be dependent on the system you set up, and may be less experienced or less capable. If they inherit a patch management system that basically entails, "Become a security expert in addition to your other duties," they are not going to strictly adhere.
You know your system and your situation, so I don't want to sound like your approach can never ever work. But if we're giving advice to the unwashed masses, I think we need more advice tailored for the people you dismiss as "too lazy to pay attention to security," because that describes basically everyone.
PS - When you describe other people's opinions as "lazy cop outs," it can kill discussions. You might watch out for that. HN isn't Reddit, people sometimes bury stuff that has good substance but poor tone.
Please tell me you are joking.
The is no way you could possibly keep up with every vulnerability of everything installed on your server.
> If somebody breaks in and gets a local shell, all is lost anyway.
That is not true at all. You should run your server such that someone could get a shell running as the apache user - and still be able to do very little. They could read files and the database (which is bad), but not modify any files (which would be worse).
The is no way you could possibly keep up with every vulnerability of everything installed on your server.
> If somebody breaks in and gets a local shell, all is lost anyway.
That is not true at all. You should run your server such that someone could get a shell running as the apache user - and still be able to do very little. They could read files and the database (which is bad), but not modify any files (which would be worse).
Of course I can and will keep up with every vulnerability for every service that is running and facing the Internet.
I do not accept the risk of waiting for some vendor to release a patch. If there's a hole, read the report, determine whether your config/build is vulnerable, rebuild.
Why would want to patch something you are not running or use?
I do not accept the risk of waiting for some vendor to release a patch. If there's a hole, read the report, determine whether your config/build is vulnerable, rebuild.
Why would want to patch something you are not running or use?
WRT local shells, it might well be a good idea to assume someone who got a shell as apache could use a privilege escalation 0-day and do some more damage. Hopefully your deployment process is such that starting from scratch isn't a huge hardship. I'd appreciate of someone with actual security experience (not me) weighed in...
It's very naive to think that you can protect a box from somebody with a local shell. Never worked.
I can't make head or tail of this comment. Are you saying you think it's a bad idea to keep my Linux kernel and Nginx up to date? What good does it do to "rebuild ASAP" unless you've at least downloaded source updates from the developers? Or are you telling me you write your own security fixes for all the software you use in public-facing services?
Of course it's a bad idea to make unnecessary system changes (install patches) that bring system to essentially unknown state that nobody ever tested (the order and set of patches installed over your specific OS configuration).
You only patch what you need to patch. Most of the time for every production service you end up building a custom version anyway. Patching does no good to those.
So, by patching you only bring potential harm and overhead of going through change control processes.
You only patch what you need to patch. Most of the time for every production service you end up building a custom version anyway. Patching does no good to those.
So, by patching you only bring potential harm and overhead of going through change control processes.