Mach's designers simply assumed that systems would be rebooted often enough(clozure.com)
clozure.com
Mach's designers simply assumed that systems would be rebooted often enough
http://clozure.com/pipermail/openmcl-devel/2011-February/012567.html
76 comments
At my last sysadmin job, we had a policy of bouncing a server if it's uptime exceeded one year, not to clean up any resources, but to make sure the config in the files matched the config that was running. Of course, not only were we running Linux (gentoo, in fact), but a lot of our stuff was DJB services, including his service manager.
I'd argue that a 5 why's analysis of "server up too long", leads to "server wasn't written well." YMMV
I'd argue that a 5 why's analysis of "server up too long", leads to "server wasn't written well." YMMV
Richard P. Gabriel makes that point somewhere - that one of the downsides of an 'organic' system like a Lisp or Smalltalk image where you can rewrite or do anything dynamically is that you tend to do just that, and can wind up in a situation where you are no longer able to stop using the image because it has too many valuable changes which are too difficult to reverse-engineer.
That's a bit of an understatement. Basically all Squeak and VisualWorks images have been running since some time in the late '70s. They've been migrated from 16- to 32- to 64-bits and across chip architectures, but it's the same image. Other dialects were bootstrapped more recently, and some of them have retained the ability to create an image from scratch.
I guess bootstrapping is more common in the Lisp world.
I guess bootstrapping is more common in the Lisp world.
I remember believing (though without any supporting evidence) that this was one reason why SmallTalk couldn't possibly catch on commercially. The system was so mutable that any kind of support was impossible, there as no way the support person could know what your system configuration was.
Now this is an excellent reason and timeframe for doing periodic reboots.
It's basically a scheduled sanity check for your infrastructure. It boils down to robustness and assurance. If parts of your system won't survive a reboot this is an unknown and therefore a risk.
If a system can't survive a reboot it's better to find that out when it's during a scheduled reboot and personnel is on hand to deal with it rather than relying on on-call personnel responding to alarms from monitoring at some point.
Apart from the repair time being longer when relying on on-call technicians, there is also the general observation that productive systems tend to become more valuable or have more exposure over time as their usage increases. A failure at a later point therefore also becomes more costly as time goes by.
It's basically a scheduled sanity check for your infrastructure. It boils down to robustness and assurance. If parts of your system won't survive a reboot this is an unknown and therefore a risk.
If a system can't survive a reboot it's better to find that out when it's during a scheduled reboot and personnel is on hand to deal with it rather than relying on on-call personnel responding to alarms from monitoring at some point.
Apart from the repair time being longer when relying on on-call technicians, there is also the general observation that productive systems tend to become more valuable or have more exposure over time as their usage increases. A failure at a later point therefore also becomes more costly as time goes by.
> Now this is an excellent reason and timeframe for doing periodic reboots.
Most of the time you don't need to do complete reboots. Restarting services usually takes care of the majority of the sanity checks.
Most of the time you don't need to do complete reboots. Restarting services usually takes care of the majority of the sanity checks.
> Restarting services usually takes care of the majority of the sanity checks.
Sure, that usually takes care of most checks, but the point is to eliminate as much as possible unusual scenarios. A service-level restart will catch service config mismatches, but it won't catch, for example, manual interface or routing table updates. It also won't catch the 'fragile root' problem where the device that the kernel is loaded from has become corrupt.
Bad things happen in the strangest places if you run enough systems.
Sure, that usually takes care of most checks, but the point is to eliminate as much as possible unusual scenarios. A service-level restart will catch service config mismatches, but it won't catch, for example, manual interface or routing table updates. It also won't catch the 'fragile root' problem where the device that the kernel is loaded from has become corrupt.
Bad things happen in the strangest places if you run enough systems.
True. You still would need to reboot from time to time, it's just that you can restart services much more frequently, even automatically if your humans know what is going down today and what can they expect if things go wrong.
But I agree - the objective is to detect those situations when things have gone wrong a long time ago and we just don't know it yet. ;-)
But I agree - the objective is to detect those situations when things have gone wrong a long time ago and we just don't know it yet. ;-)
Mainframe shops regularly reboot their systems on a rolling basis.
A common pattern is that each logical partition gets rebooted once per month, on a rotating basis. Say you have 4 LPARs: LPAR 1 is rebooted on the first week, LPAR 2 on the second and so on.
The goal is to demonstrate that in case of catastrophic failure that the system will come up. If it doesn't come back up during the scheduled reboot, you have the advantage that you are rebooting outside production hours and have some time to fix the problem minus frantic phonecalls.
Meanwhile, the other LPARs will keep running as if nothing is going on.
A common pattern is that each logical partition gets rebooted once per month, on a rotating basis. Say you have 4 LPARs: LPAR 1 is rebooted on the first week, LPAR 2 on the second and so on.
The goal is to demonstrate that in case of catastrophic failure that the system will come up. If it doesn't come back up during the scheduled reboot, you have the advantage that you are rebooting outside production hours and have some time to fix the problem minus frantic phonecalls.
Meanwhile, the other LPARs will keep running as if nothing is going on.
Yahoo reboots IM servers daily. Easier than bothering to find all the horrible leaks.
Its a business decision. Not worth getting emotionally involved, just weight cost of fixing vs cost of rebooting.
Its a business decision. Not worth getting emotionally involved, just weight cost of fixing vs cost of rebooting.
See that's not actually solving the problem. Thats more like pretending that the problem doesn't actually exist. Instead of investigate and fix permanently so that customers don't experience problems they'd rather just keep letting the problems fester.
Not everyone is "pretending that the problem doesn't actually exist." There are limited resources available to any and every organization. Many times, a proper fix can cost more than a band aid, sometimes even over time.
Here's a hypothetical, but not ridiculous, scenario. Which is a better way to spend resources? Track down and stamp out an extremely difficult resource leakage bug? Or simply bounce the server? The costs associated with the former may be HUGE. The costs associated with the latter may be small in comparison (cost of potential down time, minimal labor cost of bouncing the server(s), and costs associated with user reliability).
I wouldn't always chalk it up to whatever it is that your comment implies (laziness? willful ignorance?).
Here's a hypothetical, but not ridiculous, scenario. Which is a better way to spend resources? Track down and stamp out an extremely difficult resource leakage bug? Or simply bounce the server? The costs associated with the former may be HUGE. The costs associated with the latter may be small in comparison (cost of potential down time, minimal labor cost of bouncing the server(s), and costs associated with user reliability).
I wouldn't always chalk it up to whatever it is that your comment implies (laziness? willful ignorance?).
Yeah I'm absolutely implying something by my comment. I understand the time:value relationship and can expect some degree of RCA to go away if rebooting a server solves a problem for X amount of time, where X is less than a few days. But if you're spending the time opening an alarm, calling people to a bridge, agreeing to bounce a server, and then bouncing it, at some point that equation comes out in favor of actually doing some sysadmin/RCA and making it so you're not bouncing a server every 48/72 hours.
It also looks really JV to see those alerts day in/day out. Like, come on. We can't just fix the problem?
It also looks really JV to see those alerts day in/day out. Like, come on. We can't just fix the problem?
I really doubt anybody would accept that. The solution in many cases are simply to reboot the servers every night.
Sure, we would all love to fix the bugs, but if there are no easy clues and a regular reboot fixes it - who really cares when there are features to build?
Sure, we would all love to fix the bugs, but if there are no easy clues and a regular reboot fixes it - who really cares when there are features to build?
[deleted]
[deleted]
Mach is a microkernel. That means it's a way to design an OS such that most of the things application programs rely on are outside the kernel, provided by servers that can be restarted and replaced without rebooting the whole system. For example, the majority of the filesystem is provided by a filesystem server, most of the networking code is in a networking server, and so on.
You begin to see why a microkernel that needs to be bounced more often than, say, Linux, which is not a microkernel, begins to lose some of the appeal of having a microkernel in the first place.
(Aside: Mac OS X is based on Mach, but practically all of the stuff applications see is provided by a single server which (to the best of my knowledge) can't be restarted without taking down the whole system. It's a hybrid design.)
Also, just to confirm a stereotype, do you use Windows for your servers?
You begin to see why a microkernel that needs to be bounced more often than, say, Linux, which is not a microkernel, begins to lose some of the appeal of having a microkernel in the first place.
(Aside: Mac OS X is based on Mach, but practically all of the stuff applications see is provided by a single server which (to the best of my knowledge) can't be restarted without taking down the whole system. It's a hybrid design.)
Also, just to confirm a stereotype, do you use Windows for your servers?
Also, just to confirm a stereotype, do you use Windows for your servers?
Related tangent: There was a time when the the US military's COTS (Civilian Off The Shelf) initiative resulted in AEGIS missile cruisers running largely on machines running Windows NT. Yes, the ships responsible for screening carrier task forces against attack -- had to be rebooted on a weekly schedule.
To be fair, the unmolested NT kernel is rock solid. I know people used to run RAID arrays with it, but for the desktop, Microsoft allowed the crossing of certain architectural lines.
Related tangent: There was a time when the the US military's COTS (Civilian Off The Shelf) initiative resulted in AEGIS missile cruisers running largely on machines running Windows NT. Yes, the ships responsible for screening carrier task forces against attack -- had to be rebooted on a weekly schedule.
To be fair, the unmolested NT kernel is rock solid. I know people used to run RAID arrays with it, but for the desktop, Microsoft allowed the crossing of certain architectural lines.
And if the NT machines running your missile cruiser don't come back up, then you have real problems. During trials, the USS Yorktown had to be repeatedly towed into port after systems failures...
http://lists.essential.org/1998/am-info/msg03829.html
http://lists.essential.org/1998/am-info/msg03829.html
From what I understand the NT kernel was rock solid, if a little picky about hardware. The problem was that graphics performance suffered due to the strict isolation of the drivers from the kernel. MS relaxed those restrictions in later versions of their OS (either NT4 or Windows 2000), gaining better graphics performance at the cost of kernel stability.
>the unmolested NT kernel is rock solid.
which, if memory serves me right, directly translates to "OS/2 kernel is rock solid"
which, if memory serves me right, directly translates to "OS/2 kernel is rock solid"
No, I don't think NT is a derivative of OS/2.
"Given these issues [with OS/2], Microsoft started to work in parallel on a version of Windows which was more future-oriented and more portable. The hiring of Dave Cutler, former VMS architect, in 1988 created an immediate competition with the OS/2 team, as Cutler did not think much of the OS/2 technology and wanted to build on his work at Digital rather than creating a "DOS plus". His "NT OS/2," was a completely new architecture." http://en.wikipedia.org/wiki/OS/2
"Given these issues [with OS/2], Microsoft started to work in parallel on a version of Windows which was more future-oriented and more portable. The hiring of Dave Cutler, former VMS architect, in 1988 created an immediate competition with the OS/2 team, as Cutler did not think much of the OS/2 technology and wanted to build on his work at Digital rather than creating a "DOS plus". His "NT OS/2," was a completely new architecture." http://en.wikipedia.org/wiki/OS/2
[deleted]
Also closely related to "VAX kernel is rock solid"
Windows NT is more closely related to VMS than OS/2, but most directly related to DEC's unreleased Mica OS for their unreleased PRISM CPU architecture.
This is unsurprising: They're all designed by the same guy, Dave Cutler.
This is unsurprising: They're all designed by the same guy, Dave Cutler.
> the unmolested NT kernel is rock solid
Too bad it resents running programs on it so much...
Too bad it resents running programs on it so much...
It's not that. It's more boneheaded stuff like injecting components supporting the GUI into the kernel.
When that happens, it doesn't resent those programs so much as that it's been violated and turned into a warped version of itself. Sort of like Jeff Goldblum's character in The Fly.
When that happens, it doesn't resent those programs so much as that it's been violated and turned into a warped version of itself. Sort of like Jeff Goldblum's character in The Fly.
Mach isn't a microkernel. For example, NeXTSTEP was based on Mach 2.5, which is a monolithic kernel. Mach 3.0 was the only microkernel version, and to my knowledge, no one has adopted it with any success.
Mach 3.0 was folded into XNU in roughly the same way that earlier versions of Mach were used, but with more of the BSD functionality ported down into Mach rather than hosted POE-style in kernel space.
So 3.0 was adopted, with success, but not as a microkernal hosting user-space servers.
So 3.0 was adopted, with success, but not as a microkernal hosting user-space servers.
Does "POE" stand for something? POSIX Operating Environment, maybe? A quick Googling and a look at the Mach Wikipedia page doesn't reveal the answer.
Thanks for the great explanation, too, btw.
Hmm, nope. Solaris of some flavor I think.
Oh boy...
That's what you get when you assign the management of Solaris servers to MCSAs...
That's what you get when you assign the management of Solaris servers to MCSAs...
So if windows systems are unreliable it's windows fault, and if other OSs are unreliable it's also windows fault?
No. It's just that Solaris (and every other kind of Unix) admins know very well that they don't bounce servers when they don't know what's wrong - they find out what's causing the problem and correct it. There is a very rich set of tools to make that easy.
Bouncing a server may even make the problem go away, but it does little to enlighten you on why it happened in the first place.
And yes, If your MCSAs tend to bounce servers all the time, that's because their experiences have shown this works for Windows.
Bouncing a server may even make the problem go away, but it does little to enlighten you on why it happened in the first place.
And yes, If your MCSAs tend to bounce servers all the time, that's because their experiences have shown this works for Windows.
There are plenty of tools for troubleshooting problems on windows. And your statement that "every kind of unix admin knows not to do that" seems plainly at odds with the reality here.
> seems plainly at odds with the reality here.
You have a human resources problem, not a technical one.
You have a human resources problem, not a technical one.
I wonder if that's been fixed in the OS X/Darwin version of Mach, seems like the only time I ever reboot my MacBook is for system updates.
While the discussion in that thread may or may not apply to OS X, be very careful when saying "X is true about Mach therefore X is probably true about Darwin". A friend of mine put it very well:
OS X was Mach once in the same way that orcs were elves once.
OS X was Mach once in the same way that orcs were elves once.
The original poster in that thread was running 10.5.8. No one in the thread posted saying that they couldn't repro the port leak. The thread is quite recent, the latest post being from earlier today.
I think we have to assume that the port leak is still there in the most recent version (10.6.6).
I do hope that the poster had misunderstood Avie Tevanian in his conversations with him. Tevanian was one of Apple's chief architects and was one of the big wheels responsible for NextStep's transformation to OS X. It would be kind of outrageous for an OS architect to believe that it's at all acceptable to allow user-mode processes to cause kernel resource leaks.
I think we have to assume that the port leak is still there in the most recent version (10.6.6).
I do hope that the poster had misunderstood Avie Tevanian in his conversations with him. Tevanian was one of Apple's chief architects and was one of the big wheels responsible for NextStep's transformation to OS X. It would be kind of outrageous for an OS architect to believe that it's at all acceptable to allow user-mode processes to cause kernel resource leaks.
It was an engineering trade-off like any other. Would you be willing to reboot your computer once a month if it meant everything would be 1000x faster? Of course. So were the Mach designers, apparently.
You're misinterpreting the post; Mikel wasn't talking about the same leak the original thread was. He was just talking about how he fixed a leak many years ago.
Using Mach ports correctly without leaking is hard, similarly to how using C's malloc/free without leaking is hard. The former no more implies a current flaw in Mach than the latter implies a current flaw in C stdlib.
Using Mach ports correctly without leaking is hard, similarly to how using C's malloc/free without leaking is hard. The former no more implies a current flaw in Mach than the latter implies a current flaw in C stdlib.
Kernels should not have resource leaks. A web server that leaks memory implies nothing wrong with the clients, a kernel that leaks memory implies nothing wrong with the user space programs.
It just means that there are cases that the kernel developers didn't think of, just like the people who made Apache didn't think of the Slowloris attack. Heck, a few months ago someone exhausted Linux kernel fds in just by shuffling socket pairs around (just 40 lines of C with three syscalls).
These are problems, ranging from remote DOS attacks, local DOS attacks, to accidental DOS attacks.
It just means that there are cases that the kernel developers didn't think of, just like the people who made Apache didn't think of the Slowloris attack. Heck, a few months ago someone exhausted Linux kernel fds in just by shuffling socket pairs around (just 40 lines of C with three syscalls).
These are problems, ranging from remote DOS attacks, local DOS attacks, to accidental DOS attacks.
I wonder if there's an update for my MkLinux box!
http://mklinux.org/
http://mklinux.org/
This: the only time I ever reboot my MacBook is for system updates
Seems to indicate that it has ;-)
Seems to indicate that it has ;-)
really man? maybe you are not using your macbook long enough or you are simply browsing the internet with it?
I reboot my macbook pro (running leopard) as often as I reboot my windows 7 PC which is not much. Except I use the Win7 machine 90% of the time and for much more "intensive" stuff.
I reboot my macbook pro (running leopard) as often as I reboot my windows 7 PC which is not much. Except I use the Win7 machine 90% of the time and for much more "intensive" stuff.
I use it probably 14+ hours a day, and it's variously running MySQL, Redis, MongoDB, Photoshop, 600 million Chrome tabs, Xcode and Starcraft 2 whenever possible :)
I did experience memory leaks with pretty much every Bittorrent client, but I haven't torrented any uh, Linux distros in forever.
I did experience memory leaks with pretty much every Bittorrent client, but I haven't torrented any uh, Linux distros in forever.
Just asking, but how do you keep so many Chrome tabs open without things getting sluggish? My MBP (2008 pre-unibody/Leopard) seems to get bogged down with just a couple score of Chrome tabs.
Granted, I don't know much about what's going on under the hood (I'm a designer, not a programmer) but Activity Monitor shows that each tab uses a gig of virtual memory...
Granted, I don't know much about what's going on under the hood (I'm a designer, not a programmer) but Activity Monitor shows that each tab uses a gig of virtual memory...
Chrome's multiprocess architecture is brilliant on paper, but I've had more problems with it in practice than I care to list. Put simply: it's not a good idea to launch a separate process for each tab. If it was, nobody would use threads for anything and everyone would be happy.
This is what used to happen to Chrome on my MacBook when I had "only" 2GB of RAM:
1. Navigate to HN and click on 10 interesting stories + related comment threads.
2. Chrome tries to launch 20 processes at once. Processor usage goes through the roof and knocks several weather satellites out of their orbits.
3. The entire OS X UI freezes. I can't even Cmd+Tab to another application.
4. Some Chrome processes inexplicably crash (why?!). Nearly every other app gets swapped to disk.
5. Some old tabs get swapped to disk (which means switching to them later takes 3 to 5 seconds).
6. Processor usage becomes normal after 2 minutes. The OS becomes responsive again.
7. All my frustrated Cmd+Tabs and mouse clicks are finally registered and the UI does a halting, jitery dance as my inputs are de-queued and routed to several different apps.
8. After figuring out what buttons I accidentally clicked and in which apps, I can read HN!
Note: I know I've done a fair bit of Chrome-bashing on HN lately. If that bugs you, I apologize. It's just that I can't stand terribly engineered software grabbing a large share of the market. Chrome makes some interesting trade-offs re efficiency and the result is really not that bad. Chrome works fine for most non-geek, non-tab-happy people. For people who pretty much live in their browsers, though, Chrome is a terrible choice. Unless, of course, you have powerful machines. I believe Chrome's multiprocess model will be less of a bottleneck as processors evolve, but it's a pain in the ass on most contemporary machines.
This is what used to happen to Chrome on my MacBook when I had "only" 2GB of RAM:
1. Navigate to HN and click on 10 interesting stories + related comment threads.
2. Chrome tries to launch 20 processes at once. Processor usage goes through the roof and knocks several weather satellites out of their orbits.
3. The entire OS X UI freezes. I can't even Cmd+Tab to another application.
4. Some Chrome processes inexplicably crash (why?!). Nearly every other app gets swapped to disk.
5. Some old tabs get swapped to disk (which means switching to them later takes 3 to 5 seconds).
6. Processor usage becomes normal after 2 minutes. The OS becomes responsive again.
7. All my frustrated Cmd+Tabs and mouse clicks are finally registered and the UI does a halting, jitery dance as my inputs are de-queued and routed to several different apps.
8. After figuring out what buttons I accidentally clicked and in which apps, I can read HN!
Note: I know I've done a fair bit of Chrome-bashing on HN lately. If that bugs you, I apologize. It's just that I can't stand terribly engineered software grabbing a large share of the market. Chrome makes some interesting trade-offs re efficiency and the result is really not that bad. Chrome works fine for most non-geek, non-tab-happy people. For people who pretty much live in their browsers, though, Chrome is a terrible choice. Unless, of course, you have powerful machines. I believe Chrome's multiprocess model will be less of a bottleneck as processors evolve, but it's a pain in the ass on most contemporary machines.
If the user is quickly opening tabs, then the browser should probably use some QoS heuristic to queue or delay loading of tabs.
The Firefox "BarTab" extension does something like this:
https://addons.mozilla.org/en-us/firefox/addon/bartab/
The Firefox "BarTab" extension does something like this:
https://addons.mozilla.org/en-us/firefox/addon/bartab/
Virtual Memory consumption usually isn't worth paying attention to if you don't understand what's going on on a technical level, because it's meaningless for most purposes (each tab could be sharing 99% of that gig of address space, and that address space may never need to be mapped into real RAM, etc).
Just a guess, I think there might be some APIs available in SnowLeopard that allow Chrome to write its tabbed content to files and read it in asynchronously more easily than they could have in Leopard. Sometimes when I switch to a tab that has been idle for over a hour, I notice it takes chrome some time to render it again.
Also, most developers tend to buy machines with the maximum RAM they can. I've got 4GB in my macbook pro and the 20 chrome tabs can't be taking up more than 500MB all together. Comparatively, Safari seems to take up a lot more memory than Chrome.
Also, most developers tend to buy machines with the maximum RAM they can. I've got 4GB in my macbook pro and the 20 chrome tabs can't be taking up more than 500MB all together. Comparatively, Safari seems to take up a lot more memory than Chrome.
> Sometimes when I switch to a tab that has been idle for over a hour, I notice it takes chrome some time to render it again.
That sounds like paging in effect, which is decades old. Maybe you just notice it more now!
That sounds like paging in effect, which is decades old. Maybe you just notice it more now!
Flash Block and AdBlock go a long way.
As a fellow mac user: If you are rebooting your Mac for reasons other then system upgrade, something is wrong with your machine. I say this as someone who compiles from source and runs server software on my mac almost every day. mogodb, apache, nodejs, mysql, tomcat is just the tip of the iceberg in the server department.
> If you are rebooting your Mac for reasons other then system upgrade, something is wrong with your machine
Indeed; that is the point of the article. =)
Indeed; that is the point of the article. =)
Or you have made the grave, grave mistake of installing Adobe products on your machine.
Maybe app users experience this. I use a Mac Mini for development/test as an alternate platform.
I have to reboot the dang thing every day or two. The development tools hang and screw up. The network drops and needs to be unplugged/plugged back in or rebooted. The debugger totally fails to breakpoint and only a reboot will fix it.
I alternate between banging my head on the wall, and white-hot fury at the thing. Fortunately I only occasionally have to verify the code still runs there, and don't have to live in that environment.
I have to reboot the dang thing every day or two. The development tools hang and screw up. The network drops and needs to be unplugged/plugged back in or rebooted. The debugger totally fails to breakpoint and only a reboot will fix it.
I alternate between banging my head on the wall, and white-hot fury at the thing. Fortunately I only occasionally have to verify the code still runs there, and don't have to live in that environment.
Except for those who rely on Boot Camp for gaming. These people have to reboot whenever they want to switch from work mode to game mode.
My favorite Linux bug (since fixed):
https://bugzilla.redhat.com/show_bug.cgi?id=97373 (System UPTIME reported incorrectly):
"Steps to Reproduce: 1. Boot Linux system; 2. Go away for 497 days; 3. check uptime"
https://bugzilla.redhat.com/show_bug.cgi?id=97373 (System UPTIME reported incorrectly):
"Steps to Reproduce: 1. Boot Linux system; 2. Go away for 497 days; 3. check uptime"
Take it off the queue and go on a 16 month sabbatical to somewhere really beautiful. Come back, check uptime, report to your boss that you've recreated it and you'll start digging in the code.
http://xkcd.com/583/
http://xkcd.com/583/
Relevant: http://www.usenix.org/event/osdi04/tech/full_papers/candea/c...
Microreboot – A Technique for Cheap Recovery
"A significant fraction of software failures in large-scale Internet systems are cured by rebooting, even when the exact failure causes are unknown. However, rebooting can be expensive, causing nontrivial service disruption or downtime even when clusters and failover are employed. In this work we use separation of process recovery from data recovery to enable microrebooting – a fine-grain technique for surgically recovering faulty application components, without disturbing the rest of the application."
Microreboot – A Technique for Cheap Recovery
"A significant fraction of software failures in large-scale Internet systems are cured by rebooting, even when the exact failure causes are unknown. However, rebooting can be expensive, causing nontrivial service disruption or downtime even when clusters and failover are employed. In this work we use separation of process recovery from data recovery to enable microrebooting – a fine-grain technique for surgically recovering faulty application components, without disturbing the rest of the application."
"Hello IT, have you tried turning it off and on again?"
There are some great reasons due to memory fragmentation and other issues to reboot your systems every two or three days anyway. Plus, if you pride yourself on having failover, DR systems, etc, rebooting lets you know that your failover systems are working.
Root cause seems to be "server up too long."