Canonical Livepatch(blog.dustinkirkland.com)
blog.dustinkirkland.com
Canonical Livepatch
http://blog.dustinkirkland.com/2016/10/canonical-livepatch.html
16 comments
> Oracle absorbed the kSplice thing though, and I assume the IP around it -- so unless this does something different to that general procedure, then is there a possibility of oracle doing their usual crap here?
The live patching stuff is part of kernel core https://lkml.org/lkml/2014/11/6/387
The Ubuntu proprietary stuff is creating the patches and distributing them.
The live patching stuff is part of kernel core https://lkml.org/lkml/2014/11/6/387
The Ubuntu proprietary stuff is creating the patches and distributing them.
That's a bit critical, isn't it?
Just think of servers. You want to reboot them as seldom as possible, to be online. But you also want to have critical security patches (or well, just all security patches) applied as fast as possible. It is already very easy to run unattended upgrades, and a common practice. Maybe not in a high-class startup or big enterprise, but everywhere else. With livepatching the kernel you get the kernel security patches applied immediately, which is very very useful and will make the linux server more secure in general. And even if your security practices are perfect, livepatching is still (potentially) easier and faster and thus a good addition.
> if your situation is so bad that a kernel update is the only solution
I don't get that part, btw. In every situation you always need to update the kernel when security issues are detected.
Just think of servers. You want to reboot them as seldom as possible, to be online. But you also want to have critical security patches (or well, just all security patches) applied as fast as possible. It is already very easy to run unattended upgrades, and a common practice. Maybe not in a high-class startup or big enterprise, but everywhere else. With livepatching the kernel you get the kernel security patches applied immediately, which is very very useful and will make the linux server more secure in general. And even if your security practices are perfect, livepatching is still (potentially) easier and faster and thus a good addition.
> if your situation is so bad that a kernel update is the only solution
I don't get that part, btw. In every situation you always need to update the kernel when security issues are detected.
That's totally correct, and I can't disagree with you..
If you architect things properly, then rebooting any one component shouldn't be a problem.. In such a case though, there is no need for this tool..
In practice, though, things really don't happen all nice like that. I've got loads of internal boxes with holes. Terrible? Yeah, absolutely. We're all ashamed. These aren't a real problem security wise though, if you take a second to think about it...
Say I have a box I can't bounce (or rather, won't bounce) to get a patch for some bug -- probably it'll be some DB server in a legacy env, or something so fragile that there's a massive risk in bouncing it, vs the risk of some DOS attack which could only happen from inside the network then usually, we just won't do it... Sure, that's terrible practice and not something anyone should admit, but it's the reality...
Environments like this, the attack surface is the JVM and OpenSSH..
Assuming an attacker can get through our CDN, then through our rproxies (which we do seriously patch), after that blindly break out of a JVM on an ASLR'd linux box, THEN look to pwn a kernel hole on the DB server we've not patched, then do you really think patching is going to save you?
It's a trade off; and it's all tech debt many teams are working to completely replace -- this stuff is horrible, but sometimes you have to put up with it until you can replace it..
To me it seems the attack surface isn't really the OS anymore -- it's the app. More likely whoever is hitting you is going to rip your DB off via some dodgy post sanitation bug than remotely pwning your internal boxes with kernel exploits -- so this does little to protect anyone..
If you architect things properly, then rebooting any one component shouldn't be a problem.. In such a case though, there is no need for this tool..
In practice, though, things really don't happen all nice like that. I've got loads of internal boxes with holes. Terrible? Yeah, absolutely. We're all ashamed. These aren't a real problem security wise though, if you take a second to think about it...
Say I have a box I can't bounce (or rather, won't bounce) to get a patch for some bug -- probably it'll be some DB server in a legacy env, or something so fragile that there's a massive risk in bouncing it, vs the risk of some DOS attack which could only happen from inside the network then usually, we just won't do it... Sure, that's terrible practice and not something anyone should admit, but it's the reality...
Environments like this, the attack surface is the JVM and OpenSSH..
Assuming an attacker can get through our CDN, then through our rproxies (which we do seriously patch), after that blindly break out of a JVM on an ASLR'd linux box, THEN look to pwn a kernel hole on the DB server we've not patched, then do you really think patching is going to save you?
It's a trade off; and it's all tech debt many teams are working to completely replace -- this stuff is horrible, but sometimes you have to put up with it until you can replace it..
To me it seems the attack surface isn't really the OS anymore -- it's the app. More likely whoever is hitting you is going to rip your DB off via some dodgy post sanitation bug than remotely pwning your internal boxes with kernel exploits -- so this does little to protect anyone..
> I don't get that part, btw. In every situation you always need to update the kernel when security issues are detected.
Not the GP, but he means that if you can't take downtime on a server to reboot with an updated kernel, then you have bigger problems than the kernel not being up-to-date. Because instead of a scheduled reboot, the server could crash, lose its disks, lose its power, etc. And if you can't take a scheduled reboot, how do you take those?
Not the GP, but he means that if you can't take downtime on a server to reboot with an updated kernel, then you have bigger problems than the kernel not being up-to-date. Because instead of a scheduled reboot, the server could crash, lose its disks, lose its power, etc. And if you can't take a scheduled reboot, how do you take those?
Sun tried to do it for Solaris in the 80's, Microsoft tried to do it for Vista, and they both failed, so it might be a little more complicated than "modify whatever the kernel equiv of the GOT/PLT is" (hint: if you don't even know the names of things, would it be possible for the problem to be a bit more involved than you realize?)
Remember when Heartbleed came out, and the ensuing panic if you hadn't gotten advanced disclosure?
Instead of that, how about if all your Linux servers weren't vulnerable but in exchange, you give money to Oracle. Ksplice now patches userspace as well.
Remember when Heartbleed came out, and the ensuing panic if you hadn't gotten advanced disclosure?
Instead of that, how about if all your Linux servers weren't vulnerable but in exchange, you give money to Oracle. Ksplice now patches userspace as well.
Sure the technical details are more involved, I wasn't trying to downplay the cleverness of the tech; but the point stands -- the logic is pretty simple and AFAIK that is how kSplice works it's not black magic or anything..
Some function has an issue, a module is loaded with a replacement, and instead of calling the orig function anymore, all calls go to the one we just loaded -- via some brain surgery...
Heartbleed and userspace issues such as that are a much wider topic, but ok.. sure.. I can pay someone for easy access to updates that I can apply without downtime.. I think we're missing the point though.. Tools such as ksplice and livepatch seem to pander to environments which are unable to handle the sort of patch maintenence we've been doing as long as we've had patches available..
I've seen many of these in the wild, and what we needed to do with them wasn't patch them online but sort the infrastructure out to the point where we could handle regular maintenance..
Maybe someone really needs this sort of functionality; I struggle to think of a reason besides poor architecture though.
Some function has an issue, a module is loaded with a replacement, and instead of calling the orig function anymore, all calls go to the one we just loaded -- via some brain surgery...
Heartbleed and userspace issues such as that are a much wider topic, but ok.. sure.. I can pay someone for easy access to updates that I can apply without downtime.. I think we're missing the point though.. Tools such as ksplice and livepatch seem to pander to environments which are unable to handle the sort of patch maintenence we've been doing as long as we've had patches available..
I've seen many of these in the wild, and what we needed to do with them wasn't patch them online but sort the infrastructure out to the point where we could handle regular maintenance..
Maybe someone really needs this sort of functionality; I struggle to think of a reason besides poor architecture though.
Note that, for better or worse, this uses proprietary software. From the post:
" Q: Why does canonical-livepatch client/server have a proprietary license? A: The canonical-livepatch client is part of the Landscape family of tools available to Canonical support customers. We are enabling free access to the Canonical Livepatch Service for Ubuntu community users as a mark of our appreciation for the broader Ubuntu community, and in exchange for occasional, automatic canary testing."
" Q: Why does canonical-livepatch client/server have a proprietary license? A: The canonical-livepatch client is part of the Landscape family of tools available to Canonical support customers. We are enabling free access to the Canonical Livepatch Service for Ubuntu community users as a mark of our appreciation for the broader Ubuntu community, and in exchange for occasional, automatic canary testing."
This is super neat but since I've moved my Linux servers to virtual I don't worry about rebooting for patches. The virtual Linux boxes generally reboot in less than 30s (thank you, systemd). It usually doesn't even register as a blip on my monitoring software. When I have to reboot the host, that's a 5–10 minute ordeal while the BIOS/UEFI and storage controllers initialize.
I still don't understand why they want to bypass APT here. I saw the FAQ about this and it still doesn't make sense to me and just feels like some kind of a lock-in attempt.
You can continue to use apt and just reboot when your kernel package is updated.
If you don't want to reboot, then you can add livepatching.
It works with the existing kernel packages to fill the gap before you reboot. In other words, it's not a bypass for apt; it works in addition to it.
But livepatching doesn't cover all situations. Currently it's only high priority CVEs, and the article talks of situations where it's not possible at all. Given that apt is the fallback, I don't see how anyone could consider it to be lock-in.
If you don't want to reboot, then you can add livepatching.
It works with the existing kernel packages to fill the gap before you reboot. In other words, it's not a bypass for apt; it works in addition to it.
But livepatching doesn't cover all situations. Currently it's only high priority CVEs, and the article talks of situations where it's not possible at all. Given that apt is the fallback, I don't see how anyone could consider it to be lock-in.
There are a couple pieces to that, not the least of which is Canonical wanting to push their new 'snap' format, for better or worse. The other thing is that Apt is not a good fit for the task - say it's daemonized programs instead of live patches. Apt only takes responsibility for downloading the bits onto the machine and unpacking them, and init/systemd is responsible for actually running programs.
However, for live patches, it's critical the program be running/the patch installed, and Apt doesn't manage running processes, nor does it have a UI whos purpose is to report on patches that have successfully downloaded but fail to install. These aren't fatal problems, but those aren't features upstream apt is likely to want (maybe systemd wants to swallow apt), so then you're forking apt, with all the work that entails. Might as well just write your own thing in that case.
However, for live patches, it's critical the program be running/the patch installed, and Apt doesn't manage running processes, nor does it have a UI whos purpose is to report on patches that have successfully downloaded but fail to install. These aren't fatal problems, but those aren't features upstream apt is likely to want (maybe systemd wants to swallow apt), so then you're forking apt, with all the work that entails. Might as well just write your own thing in that case.
But APT does. It can run the post-install script for the package, which can certainly include the live-patching bits. It's not like live-patching needs a continuously running daemon.
Saying that APT can't do this is nearly equivalent to saying that APT should not be trusted to run any post-install scripts at all, which basically means it shouldn't be trusted to install half the software out there.
Saying that APT can't do this is nearly equivalent to saying that APT should not be trusted to run any post-install scripts at all, which basically means it shouldn't be trusted to install half the software out there.
> maybe systemd wants to swallow apt
Say hello to Flatpak...
Say hello to Flatpak...
It sounds hardcore, but this isn't really that complicated is it? Load a module with a patched function and then modify whatever the kernel equiv of the GOT/PLT is, and boom -- hotpatch... Right?
Oracle absorbed the kSplice thing though, and I assume the IP around it -- so unless this does something different to that general procedure, then is there a possibility of oracle doing their usual crap here?
This is more of a toy than anything though, I recon...
In all my years of adopting and trying to unfuck horribly outdated systems (not by choice) -- I've never once really wanted or needed to hotpatch a kernel....
You don't solve your poor architecture decisions by hotpatching running kernels... If your situation is so bad that a kernel update is the only solution, and you can't take a box down to do that without an outage on the platform -- then you need to rethink what you're doing or hire someone who can do that for you..