Why I Won't Use Rimuhosting(aaronsw.com)
aaronsw.com
Why I Won't Use Rimuhosting
http://www.aaronsw.com/weblog/rimuhosting
18 comments
The longer you stay with a provider, the more likely you are to find a tech having a bad day ...
I'm sorry, but the behavior described in the article is completely unacceptable. I don't care how bad of a day anyone's having, you don't break into your customer's server and shut off Apache without notice.
I'm sorry, but the behavior described in the article is completely unacceptable. I don't care how bad of a day anyone's having, you don't break into your customer's server and shut off Apache without notice.
It's not "breaking in". They own the server, and pretty much every host's terms of service allow them to do pretty much anything they deem necessary to maintain service for their other customers, or in response to abuse or law enforcement requests (read the terms of service at your favorite hosting provider; you'll find that you've agreed to this kind of intervention when you signed up).
So, this particular event shows a lack of good judgment on the part of the tech in question, and a lack of common courtesy in not notifying the customer of the issue in advance of taking action...but it is not an illicit act. No "break in" happened here. The options a hosting provider has when things are behaving badly are pretty much all things that a customer will not like. When a dedicated server is compromised and is being used to send spam, for example, the host will simply firewall the whole thing off from the world, shutting down all services. This happens all the time; the effected customer blames the host, sometimes. Sometimes there are other options. In this case, the host opted to turn off one (disputably) offending service rather than kill the whole host. I think we can all agree this wasn't the best way to handle it. Thus my assertion that this is a tech having a bad day (or simply one bad support tech at Rimuhosting, among mostly good ones).
Actually, I think the one thing I'm uncomfortable with is the illusion of propriety (ownership) Rimuhosting has given by saying "You are able to remove that if you are not comfortable with that.", in reference to the ssh key. This is a bit of a disconnect from the reality that under some circumstances, they will have to exert their ownership rights, and there's nothing you can do about it. At some point you just have to trust in the good faith of your provider. But the only alternative to doing what they did (assuming they, in good faith, thought Aaron's VM was damaging service for others), assuming they don't have the ability to throttle Xen instances on this particular machine (maybe it's too old to have that capability, I dunno, or maybe the tech in question didn't actually understand those capabilities; the tech could be new...everybody starts somewhere), would be to turn his VM off completely. From the techs perspective he may have thought he was doing Aaron a favor by leaving mail and other services running.
Again, I'm not arguing it was good judgment on the part of the tech. But to say that Rimuhosting broke into their own machine is ridiculous.
So, this particular event shows a lack of good judgment on the part of the tech in question, and a lack of common courtesy in not notifying the customer of the issue in advance of taking action...but it is not an illicit act. No "break in" happened here. The options a hosting provider has when things are behaving badly are pretty much all things that a customer will not like. When a dedicated server is compromised and is being used to send spam, for example, the host will simply firewall the whole thing off from the world, shutting down all services. This happens all the time; the effected customer blames the host, sometimes. Sometimes there are other options. In this case, the host opted to turn off one (disputably) offending service rather than kill the whole host. I think we can all agree this wasn't the best way to handle it. Thus my assertion that this is a tech having a bad day (or simply one bad support tech at Rimuhosting, among mostly good ones).
Actually, I think the one thing I'm uncomfortable with is the illusion of propriety (ownership) Rimuhosting has given by saying "You are able to remove that if you are not comfortable with that.", in reference to the ssh key. This is a bit of a disconnect from the reality that under some circumstances, they will have to exert their ownership rights, and there's nothing you can do about it. At some point you just have to trust in the good faith of your provider. But the only alternative to doing what they did (assuming they, in good faith, thought Aaron's VM was damaging service for others), assuming they don't have the ability to throttle Xen instances on this particular machine (maybe it's too old to have that capability, I dunno, or maybe the tech in question didn't actually understand those capabilities; the tech could be new...everybody starts somewhere), would be to turn his VM off completely. From the techs perspective he may have thought he was doing Aaron a favor by leaving mail and other services running.
Again, I'm not arguing it was good judgment on the part of the tech. But to say that Rimuhosting broke into their own machine is ridiculous.
But to say that Rimuhosting broke into their own machine is ridiculous.
I'm not sure about that. Since the VM is virtual, you could say that they don't own the VM, they own the machine the VM runs on. They can control and manipulate the VM via the container for running them (Xen in this case). But since the VM doesn't physically exist, it could be argued that they don't actually own it, they only rent server capacity to run the VM.
Now, I know that you could similarly argue that they were only reading/modifying a disk image that existed on their server, but that isn't as much fun to argue.
I think that everyone here can agree that this was a dumb move by Rimuhosting. And I'm not sure what the contractual arrangement is when you buy a VPS from them... but what they have posted online doesn't make this very clear.
The question of ownership is interesting in this era of cloud computing (and not at all clear). I think it's clear that they own the bare metal. I think it's also clear that you own your data. However, where those mix is an interesting question. At what point is their (uninvited) intervention in a customer's VM warranted outside of their normal management options in the VM container (Xen)? At what point is the intervention too much?
Does the company matter? I can imagine everyone being up in arms if Amazon tried the same thing with an EC2 instance.
Of course this could all be alleviated by using a dedicated server... but even then, if the hosting company owns the hardware, would it be okay for them to pull a hard drive, and add themselves as root? Never. Ever. But this is effectively what we're talking about. The only difference is that with a dedicated server your actions don't affect others. In a VPS environment, your actions can affect others. But I think that there are far better methods for controlling this than "breaking in" to a customer's VM.
I'm not sure about that. Since the VM is virtual, you could say that they don't own the VM, they own the machine the VM runs on. They can control and manipulate the VM via the container for running them (Xen in this case). But since the VM doesn't physically exist, it could be argued that they don't actually own it, they only rent server capacity to run the VM.
Now, I know that you could similarly argue that they were only reading/modifying a disk image that existed on their server, but that isn't as much fun to argue.
I think that everyone here can agree that this was a dumb move by Rimuhosting. And I'm not sure what the contractual arrangement is when you buy a VPS from them... but what they have posted online doesn't make this very clear.
The question of ownership is interesting in this era of cloud computing (and not at all clear). I think it's clear that they own the bare metal. I think it's also clear that you own your data. However, where those mix is an interesting question. At what point is their (uninvited) intervention in a customer's VM warranted outside of their normal management options in the VM container (Xen)? At what point is the intervention too much?
Does the company matter? I can imagine everyone being up in arms if Amazon tried the same thing with an EC2 instance.
Of course this could all be alleviated by using a dedicated server... but even then, if the hosting company owns the hardware, would it be okay for them to pull a hard drive, and add themselves as root? Never. Ever. But this is effectively what we're talking about. The only difference is that with a dedicated server your actions don't affect others. In a VPS environment, your actions can affect others. But I think that there are far better methods for controlling this than "breaking in" to a customer's VM.
If my VPS was on the same host I'd pretty much demand they do exactly that.
Actually I'd probably demand they stop the instance, so I reckon he got a free kick if they only stopped the offending service.
Actually I'd probably demand they stop the instance, so I reckon he got a free kick if they only stopped the offending service.
CPU is one of the easiest things to throttle in Xen. there's no reason why the provider should care at all that you are running as much cpu as they let you. Proper use of the credit scheduler means this happens without administrator intervention.
You can also gracefully limit disk bandwidth and network bandwidth from the dom0, without ever getting a shell in the DomU.
You can also gracefully limit disk bandwidth and network bandwidth from the dom0, without ever getting a shell in the DomU.
Yes, and this would have been the correct way to fix the situation. Gracefully reduce the available CPU for the offending VPS, and notify the customer.
Breaking in is OK as a last resort.
Breaking in is OK as a last resort.
Personally, I think freezing the domain is preferable to logging in and killing a (likely rather important) process.
But then, the discussion is academic. I've never had to shut down a domain to prevent heavy legitimate or runaway process use from harming other users.
I have shut down many domains to stop abuse, but that's a very different thing. If you want to examine a compromised disk, you are best off doing so booting off of read-only media then carefully examining the compromised partition, so you want it shut down anyhow.
But then, the discussion is academic. I've never had to shut down a domain to prevent heavy legitimate or runaway process use from harming other users.
I have shut down many domains to stop abuse, but that's a very different thing. If you want to examine a compromised disk, you are best off doing so booting off of read-only media then carefully examining the compromised partition, so you want it shut down anyhow.
I think this is actually the heart of the matter. No one (or almost no one) is arguing that a hosting provider shouldn't be able to take steps to prevent one customer from negatively impacting others.
What is at issue is whether the tech should have stopped the instance, or logged in and located and stopped the offending service. The privacy nuts among us (I'm one of them, but with a healthy dose of "hosting industry reality" thrown in for good measure) are crying foul that anyone should ever be able to login to "your" server under any circumstances. The other side of the coin is that a large percentage of hosting customers have no idea what they're doing, and rely on the host to occasionally step in and set things right.
Hosting providers tend to default to stepping in, because that's what the majority of hosting customers want and expect. Rackspace has built a ~$1 billion business on this premise. "Managed hosting" is more expensive because you get this kind of intervention (hopefully in a more positive direction most of the time) on a regular basis.
The only wrong, I think, in this case is that the tech at Rimuhosting did not respect Aaron's specific wish to not allow him to access his box. And, since I know Rimuhosting has a good reputation that spans a decade or so, I must assume it was a mistake made in good faith. It is entirely possible the tech didn't reread the full conversation about the problem, and missed the bit about "over my dead body", and just assumed Aaron would want to keep email spinning while he figured out why his website was causing problems.
To turn this into a huge political struggle is pointless, and casts aspersions on a company that is by most accounts, one of the really good ones in their industry.
What is at issue is whether the tech should have stopped the instance, or logged in and located and stopped the offending service. The privacy nuts among us (I'm one of them, but with a healthy dose of "hosting industry reality" thrown in for good measure) are crying foul that anyone should ever be able to login to "your" server under any circumstances. The other side of the coin is that a large percentage of hosting customers have no idea what they're doing, and rely on the host to occasionally step in and set things right.
Hosting providers tend to default to stepping in, because that's what the majority of hosting customers want and expect. Rackspace has built a ~$1 billion business on this premise. "Managed hosting" is more expensive because you get this kind of intervention (hopefully in a more positive direction most of the time) on a regular basis.
The only wrong, I think, in this case is that the tech at Rimuhosting did not respect Aaron's specific wish to not allow him to access his box. And, since I know Rimuhosting has a good reputation that spans a decade or so, I must assume it was a mistake made in good faith. It is entirely possible the tech didn't reread the full conversation about the problem, and missed the bit about "over my dead body", and just assumed Aaron would want to keep email spinning while he figured out why his website was causing problems.
To turn this into a huge political struggle is pointless, and casts aspersions on a company that is by most accounts, one of the really good ones in their industry.
The only wrong, I think, in this case is that the tech at Rimuhosting did not respect Aaron's specific wish to not allow him to access his box.
Do you think this is more or less wrong than simply halting the instance?
Do you think this is more or less wrong than simply halting the instance?
Do you think this is more or less wrong than simply halting the instance?
If they had reason to believe his instance was impacting other users or their network, then halting the instance (preferably after contacting him about the problem, which seems to have happened) would not have been wrong at all. So, logging in after he'd asked them not to is definitely the "more wrong" of the two options. I tend to think it was an honest mistake made by a support tech who was having a bad/busy day rather than an attempt to violate any (real or imagined) privacy pact that the company had made with Aaron.
As I've mentioned a few times here, the default expectation of hosting customers is that the host will login to find and fix problems. From the perspective of most hosting customers, that is part of the service they are paying for (and they pay more to get more of it, as in the case of RackSpace), and if it were missing they would be angry and leave. My guess is that the tech at Rimu simply didn't remember or notice that Aaron had requested no one ever login to his box.
Basically, what I'm trying to say is that, as someone that is deeply familiar with this industry, when I read the shitstorm that has been kicked up here over this (extremely minor) incident, I'm just stunned and baffled. This is seriously a tempest in a teapot, if ever I saw one.
Yes, your hosting provider should respect your privacy, to the best of their ability to do so while shepherding their shared resources responsibly to provide the best service to each of their customers. But, there is no accusation of theft of data; no reason to believe Rimuhosting acted inappropriately with the access they had; no reason to think it was done maliciously or completely without warning (Aaron admits he'd talked with them in the past about the issue); and no reason to think Rimuhosting was trying to piss anyone off.
Not being thoroughly informed about a customer's wishes is pretty much all one can accuse them of here.
If they had reason to believe his instance was impacting other users or their network, then halting the instance (preferably after contacting him about the problem, which seems to have happened) would not have been wrong at all. So, logging in after he'd asked them not to is definitely the "more wrong" of the two options. I tend to think it was an honest mistake made by a support tech who was having a bad/busy day rather than an attempt to violate any (real or imagined) privacy pact that the company had made with Aaron.
As I've mentioned a few times here, the default expectation of hosting customers is that the host will login to find and fix problems. From the perspective of most hosting customers, that is part of the service they are paying for (and they pay more to get more of it, as in the case of RackSpace), and if it were missing they would be angry and leave. My guess is that the tech at Rimu simply didn't remember or notice that Aaron had requested no one ever login to his box.
Basically, what I'm trying to say is that, as someone that is deeply familiar with this industry, when I read the shitstorm that has been kicked up here over this (extremely minor) incident, I'm just stunned and baffled. This is seriously a tempest in a teapot, if ever I saw one.
Yes, your hosting provider should respect your privacy, to the best of their ability to do so while shepherding their shared resources responsibly to provide the best service to each of their customers. But, there is no accusation of theft of data; no reason to believe Rimuhosting acted inappropriately with the access they had; no reason to think it was done maliciously or completely without warning (Aaron admits he'd talked with them in the past about the issue); and no reason to think Rimuhosting was trying to piss anyone off.
Not being thoroughly informed about a customer's wishes is pretty much all one can accuse them of here.
The only wrong, I think, in this case is that the tech at Rimuhosting did not respect Aaron's specific wish to not allow him to access his box.
The thing is, that's just Aaron that thinks that. I'm sure other customers would be delighted for tech support to access their boxes to fix problems, indeed annoyed if they didn't.
Aaron pays $20 for a cheap-ass VPS, uses too much CPU, and they go out of their way to fix the specific problem instead of just nuking his account, and he complains on some BS privacy grounds. You just can't please some people.
The thing is, that's just Aaron that thinks that. I'm sure other customers would be delighted for tech support to access their boxes to fix problems, indeed annoyed if they didn't.
Aaron pays $20 for a cheap-ass VPS, uses too much CPU, and they go out of their way to fix the specific problem instead of just nuking his account, and he complains on some BS privacy grounds. You just can't please some people.
Yeah! You tell 'em! Nobody has any grounds to complain when a specific fucking request is ignored! Who cares if the customer gets what he wants? He should be happy he's getting what thirty other people want instead!
Criminy. Did you even read what you quoted before you responded to it?
Criminy. Did you even read what you quoted before you responded to it?
You sound like a person at McDonalds. I SPECIFICALLY ASKED FOR A QUARTER POUNDER WITHOUT PICKLES AND YET I FOUND A PICKLE IN MY QUARTER POUNDER AND I WOULD LIKE TO SPEAK TO THE MANAGER
It's more like he went to McDonald's and specifically requested that they not give him a colonoscopy -- but they did so anyway.
The data loss stories I've heard, in particular, are enough to keep me awake at night.
Over four years ago, I was helping my father-in-law establish a web presence and e-mail system for his small business. Things were fine on my personal dedicated server but I decided to move their stuff to Yahoo's "proven" business service. One day, I found out everything was removed -- email accounts, web site -- for no explainable reason. The whole sordid affair was documented here:
http://billkatz.com/2005/6/Run-Dont-Walk-from-Yahoo-Small-Bu...
Over four years ago, I was helping my father-in-law establish a web presence and e-mail system for his small business. Things were fine on my personal dedicated server but I decided to move their stuff to Yahoo's "proven" business service. One day, I found out everything was removed -- email accounts, web site -- for no explainable reason. The whole sordid affair was documented here:
http://billkatz.com/2005/6/Run-Dont-Walk-from-Yahoo-Small-Bu...
Gaining root in such an underhanded manner against the client's wishes is bordering on criminal behavior. It doesn't get much worse than that.
Of course, I write about security professionally. Maybe I have a different perspective on what constitutes unacceptable behavior than you have.
Of course, I write about security professionally. Maybe I have a different perspective on what constitutes unacceptable behavior than you have.
Gaining root in such an underhanded manner against the client's wishes is bordering on criminal behavior.
No. Not even close. Rimuhosting owns the box in question. You can't possibly break into your own box.
Of course, I write about security professionally. Maybe I have a different perspective on what constitutes unacceptable behavior than you have.
I work in the hosting industry professionally. I've never been or worked at a service provider, but I've worked for the industry for 12 years as a vendor.
As a security professional, perhaps you can take the other side's view for a moment...
What would you, as a security professional who was dumped into the role of running a hosting data center, do when a box on your network (which has thousands of other boxes) has been rooted and is sending out spam at a rate of a million messages per day, even if you know that your paying customer is not responsible for sending those messages? This much traffic is impacting network performance for others, is spewing filth into the mailboxes of innocent people all over the world, and your customer is unable to stop it or cannot be reached in a reasonable time frame. What do you believe your responsibility would be in such a case?
How about if it is (innocently) DoSing other systems on the network? This happens a lot more than you'd think.
Many people here are suggesting that simply applying resource limits would solve this problem...but that exhibits a lack of understanding of how such resource limits work, and the likely end result for the customer of such resource limiting in the event something has gone wrong. To the customer, a limit on resource usage could very well also result in services becoming unavailable. If you cap CPU at 10%, and your website gets enough traffic to need 90% of the CPU, you'll only be able to serve a small percentage of your clients (and they will wait a long time). The same is true of nearly every resource limit in a hosting environment. If you are using more than your fair share (which I know Aaron has explained he was not; again, probably poor judgment and problem assessment on the part of the tech) then when the limits are imposed, you will be "turned off" for some of your users. You needed those resources to serve all of your requests...you don't have those resources anymore, so you can't serve all of those requests anymore, or if you do, you do so very slowly.
As programmers, you all know there are no silver bullet for solving hard, complex problems. Resource usage in a virtualized environment with untrusted users is a hard, complex problem. No amount of hand-waving about your "rights" makes that less true.
No. Not even close. Rimuhosting owns the box in question. You can't possibly break into your own box.
Of course, I write about security professionally. Maybe I have a different perspective on what constitutes unacceptable behavior than you have.
I work in the hosting industry professionally. I've never been or worked at a service provider, but I've worked for the industry for 12 years as a vendor.
As a security professional, perhaps you can take the other side's view for a moment...
What would you, as a security professional who was dumped into the role of running a hosting data center, do when a box on your network (which has thousands of other boxes) has been rooted and is sending out spam at a rate of a million messages per day, even if you know that your paying customer is not responsible for sending those messages? This much traffic is impacting network performance for others, is spewing filth into the mailboxes of innocent people all over the world, and your customer is unable to stop it or cannot be reached in a reasonable time frame. What do you believe your responsibility would be in such a case?
How about if it is (innocently) DoSing other systems on the network? This happens a lot more than you'd think.
Many people here are suggesting that simply applying resource limits would solve this problem...but that exhibits a lack of understanding of how such resource limits work, and the likely end result for the customer of such resource limiting in the event something has gone wrong. To the customer, a limit on resource usage could very well also result in services becoming unavailable. If you cap CPU at 10%, and your website gets enough traffic to need 90% of the CPU, you'll only be able to serve a small percentage of your clients (and they will wait a long time). The same is true of nearly every resource limit in a hosting environment. If you are using more than your fair share (which I know Aaron has explained he was not; again, probably poor judgment and problem assessment on the part of the tech) then when the limits are imposed, you will be "turned off" for some of your users. You needed those resources to serve all of your requests...you don't have those resources anymore, so you can't serve all of those requests anymore, or if you do, you do so very slowly.
As programmers, you all know there are no silver bullet for solving hard, complex problems. Resource usage in a virtualized environment with untrusted users is a hard, complex problem. No amount of hand-waving about your "rights" makes that less true.
> No. Not even close. Rimuhosting owns the box in question. You can't possibly break into your own box.
I don't think you're familiar with the common uses of "bordering on".
> What would you, as a security professional who was dumped into the role of running a hosting data center, do when a box on your network (which has thousands of other boxes) has been rooted and is sending out spam at a rate of a million messages per day, even if you know that your paying customer is not responsible for sending those messages?
I'd stop the server without gaining root against the client's wishes, immediately inform the client, and try to get things cleaned up to everyone's satisfaction (except the security cracker who rooted the box, of course). I wouldn't gain root on the box in violation of the client's obvious wishes, particularly after articulating specific policy saying that you can remove my initial root access to the box if you like, and shut down specific services that are making my life difficult for reasons entirely unrelated to the client's running of the site, then send an email later pointing out my policy that fails to mention I might do this.
Privacy is security, after all: http://blogs.techrepublic.com.com/security/?p=293
Nothing you say in your hypothetical examples in any way justifies accessing the customer's data, particularly by breaking in to the system. Shut it down? Sure, if circumstances call for that. Start meddling with server configuration and establish what amounts to a rootkit on the system (in effect, if not in technical truth) without the client's permission? That's just shady.
> No amount of hand-waving about your "rights" makes that less true.
Perhaps you should stop waving your hands about that straw man, then.
I don't think you're familiar with the common uses of "bordering on".
> What would you, as a security professional who was dumped into the role of running a hosting data center, do when a box on your network (which has thousands of other boxes) has been rooted and is sending out spam at a rate of a million messages per day, even if you know that your paying customer is not responsible for sending those messages?
I'd stop the server without gaining root against the client's wishes, immediately inform the client, and try to get things cleaned up to everyone's satisfaction (except the security cracker who rooted the box, of course). I wouldn't gain root on the box in violation of the client's obvious wishes, particularly after articulating specific policy saying that you can remove my initial root access to the box if you like, and shut down specific services that are making my life difficult for reasons entirely unrelated to the client's running of the site, then send an email later pointing out my policy that fails to mention I might do this.
Privacy is security, after all: http://blogs.techrepublic.com.com/security/?p=293
Nothing you say in your hypothetical examples in any way justifies accessing the customer's data, particularly by breaking in to the system. Shut it down? Sure, if circumstances call for that. Start meddling with server configuration and establish what amounts to a rootkit on the system (in effect, if not in technical truth) without the client's permission? That's just shady.
> No amount of hand-waving about your "rights" makes that less true.
Perhaps you should stop waving your hands about that straw man, then.
I don't think you're familiar with the common uses of "bordering on".
I'll promise to read up on it, if you'll learn the meaning of "not even close".
Nothing you say in your hypothetical examples in any way justifies accessing the customer's data, particularly by breaking in to the system. Shut it down? Sure, if circumstances call for that. Start meddling with server configuration and establish what amounts to a rootkit on the system (in effect, if not in technical truth) without the client's permission? That's just shady.
I will merely mention that most hosting provider customers not only accept this kind of thing, they demand it. If you spend a little time on web hosting forums (as I must because it is my industry) you will notice a very strong tendency for complaints to be system administration related. The customer expected more involvement than the hosting provider offered, and thus things went horribly awry.
I agree with you, entirely, that if you ask the host to never login to your system, they should respect that wish. But, I can also state without hesitation that you and I (and most people here at HN) are thoroughly in the minority in wanting our hosting provider to never login to our hosting systems. The default mode for hosting providers is to drop in on the box within a couple of comments in their ticketing system...if it can't be solved with one or two replies, then it's safest to simply drop in and fix it. For most hosting customers this is not an invasion of privacy or "breaking in", it is "great support".
Finally, as a security professional, I'm sure you're also aware that with access to the hardware, your host has root all the time. There is nothing you or I can do about it. Even more interestingly, the host also has the ability to login, poke around, and never tell you about it (and not leave a trail...just boot up a live CD, mount up the disk read only, and poke around til their heart's content). Also nothing you or I can do about that. With someone else having access to the hardware, you have nothing but good faith on the part of the hosting provider.
I'll promise to read up on it, if you'll learn the meaning of "not even close".
Nothing you say in your hypothetical examples in any way justifies accessing the customer's data, particularly by breaking in to the system. Shut it down? Sure, if circumstances call for that. Start meddling with server configuration and establish what amounts to a rootkit on the system (in effect, if not in technical truth) without the client's permission? That's just shady.
I will merely mention that most hosting provider customers not only accept this kind of thing, they demand it. If you spend a little time on web hosting forums (as I must because it is my industry) you will notice a very strong tendency for complaints to be system administration related. The customer expected more involvement than the hosting provider offered, and thus things went horribly awry.
I agree with you, entirely, that if you ask the host to never login to your system, they should respect that wish. But, I can also state without hesitation that you and I (and most people here at HN) are thoroughly in the minority in wanting our hosting provider to never login to our hosting systems. The default mode for hosting providers is to drop in on the box within a couple of comments in their ticketing system...if it can't be solved with one or two replies, then it's safest to simply drop in and fix it. For most hosting customers this is not an invasion of privacy or "breaking in", it is "great support".
Finally, as a security professional, I'm sure you're also aware that with access to the hardware, your host has root all the time. There is nothing you or I can do about it. Even more interestingly, the host also has the ability to login, poke around, and never tell you about it (and not leave a trail...just boot up a live CD, mount up the disk read only, and poke around til their heart's content). Also nothing you or I can do about that. With someone else having access to the hardware, you have nothing but good faith on the part of the hosting provider.
> I'll promise to read up on it, if you'll learn the meaning of "not even close".
I don't see how "I know you are, but what am I?" is a very strong argument.
> I will merely mention that most hosting provider customers not only accept this kind of thing, they demand it.
Give it to customers who want it. Don't ask someone if he wants it, get a "no" answer, then turn around and do it anyway. The former is good customer service. The latter is shady and underhanded.
> The default mode for hosting providers is to drop in on the box within a couple of comments in their ticketing system...if it can't be solved with one or two replies, then it's safest to simply drop in and fix it. For most hosting customers this is not an invasion of privacy or "breaking in", it is "great support".
Most hosting customers don't have explicit suggestions from the host that if they don't want the hosting provider logging in to the system they can remove their SSH keys -- and, more to the point, most hosting customers don't do that then find out the hosting provider's support personnel have been rooting around (pardon the pun) in their data anyway.
It's "breaking in" in this case only because the SSH key for access was removed, with the hosting provider's blessing, and they basically leveraged a local access vulnerability to give themselves root access.
If the hosting provider had a policy that forbade customers from obstructing host administrators from logging in to the machine, articulated in the terms and conditions of use, I'd say go for it -- but that's not the situation in this case at all.
> Finally, as a security professional, I'm sure you're also aware that with access to the hardware, your host has root all the time.
In principle, sure -- but when there's a clearly encouraged expectation of privacy, it's really bad form to break in to the system against the customer's wishes by virtue of having access to the hardware. That's a betrayal of trust.
> Even more interestingly, the host also has the ability to login, poke around, and never tell you about it (and not leave a trail...just boot up a live CD, mount up the disk read only, and poke around til their heart's content).
Indeed. The fact it wasn't kept secret in this case just shows how little they value the customer's request that they don't log in to the system with root privileges and muck about with the customer's data. I didn't say they were necessarily malicious about it -- but that doesn't mean it's not bad.
> With someone else having access to the hardware, you have nothing but good faith on the part of the hosting provider.
. . . or, in this case, bad faith.
I don't see how "I know you are, but what am I?" is a very strong argument.
> I will merely mention that most hosting provider customers not only accept this kind of thing, they demand it.
Give it to customers who want it. Don't ask someone if he wants it, get a "no" answer, then turn around and do it anyway. The former is good customer service. The latter is shady and underhanded.
> The default mode for hosting providers is to drop in on the box within a couple of comments in their ticketing system...if it can't be solved with one or two replies, then it's safest to simply drop in and fix it. For most hosting customers this is not an invasion of privacy or "breaking in", it is "great support".
Most hosting customers don't have explicit suggestions from the host that if they don't want the hosting provider logging in to the system they can remove their SSH keys -- and, more to the point, most hosting customers don't do that then find out the hosting provider's support personnel have been rooting around (pardon the pun) in their data anyway.
It's "breaking in" in this case only because the SSH key for access was removed, with the hosting provider's blessing, and they basically leveraged a local access vulnerability to give themselves root access.
If the hosting provider had a policy that forbade customers from obstructing host administrators from logging in to the machine, articulated in the terms and conditions of use, I'd say go for it -- but that's not the situation in this case at all.
> Finally, as a security professional, I'm sure you're also aware that with access to the hardware, your host has root all the time.
In principle, sure -- but when there's a clearly encouraged expectation of privacy, it's really bad form to break in to the system against the customer's wishes by virtue of having access to the hardware. That's a betrayal of trust.
> Even more interestingly, the host also has the ability to login, poke around, and never tell you about it (and not leave a trail...just boot up a live CD, mount up the disk read only, and poke around til their heart's content).
Indeed. The fact it wasn't kept secret in this case just shows how little they value the customer's request that they don't log in to the system with root privileges and muck about with the customer's data. I didn't say they were necessarily malicious about it -- but that doesn't mean it's not bad.
> With someone else having access to the hardware, you have nothing but good faith on the part of the hosting provider.
. . . or, in this case, bad faith.
xm sched-credit -d username -w 10
would have solved the CPU problem without doing anything more than slowing down the heavy user. Giving each guest a single CPU rather than all CPUs on the box also helps a lot. In fact, if your 'bad' user bought the same percentage of the box as everyone else, if you left the weights at their default levels, that user would get approximately their fair share of CPU.
Tip: if you don't dedicate a core to the Dom0, you will have problems if anyone else uses much cpu, disk, or network.
If you want to do the same for network, simply use tc: http://wiki.xen.prgmr.com/xenophilia/2008/06/traffic-shaping... - if your problem is disk, use ionice: http://wiki.xen.prgmr.com/xenophobia/2008/06/using-ionice1-t...
The whole point of providing Xen instances is that I don't want to muck around in your instance. It pisses you off, and it wastes my time. That's why I'm using xen, and not OpenVZ or some other glorified chroot. From the point of view of a provider, handling everything within the Dom0 not only makes customers more comfortable, it's also less work for you, the provider.
would have solved the CPU problem without doing anything more than slowing down the heavy user. Giving each guest a single CPU rather than all CPUs on the box also helps a lot. In fact, if your 'bad' user bought the same percentage of the box as everyone else, if you left the weights at their default levels, that user would get approximately their fair share of CPU.
Tip: if you don't dedicate a core to the Dom0, you will have problems if anyone else uses much cpu, disk, or network.
If you want to do the same for network, simply use tc: http://wiki.xen.prgmr.com/xenophilia/2008/06/traffic-shaping... - if your problem is disk, use ionice: http://wiki.xen.prgmr.com/xenophobia/2008/06/using-ionice1-t...
The whole point of providing Xen instances is that I don't want to muck around in your instance. It pisses you off, and it wastes my time. That's why I'm using xen, and not OpenVZ or some other glorified chroot. From the point of view of a provider, handling everything within the Dom0 not only makes customers more comfortable, it's also less work for you, the provider.
You can't possibly break into your own box.
Yes you can; I've broken into my own house before.
Personally, I'd like it if they gave you notice before doing anything to a VPS you may be using so that you have the opportunity to shut things down gracefully.
Even a sudden CPU throttle would be better than shutting down services on the customer's VPS.
Edit: spelling
Yes you can; I've broken into my own house before.
Personally, I'd like it if they gave you notice before doing anything to a VPS you may be using so that you have the opportunity to shut things down gracefully.
Even a sudden CPU throttle would be better than shutting down services on the customer's VPS.
Edit: spelling
as another security professional I'm inclined to agree with Swelljoe.
(mostly because it's clearly going to be in the T&C's)
(mostly because it's clearly going to be in the T&C's)
If there's something in the terms and conditions that says they reserve the right to change the configuration on your system because their explicitly identified policies for dealing with high CPU usage aren't working, I might back off from the "borderline criminal" descriptor -- but I'd still say that's a great way to make customers feel violated and run like the wind to other service providers. As many others have pointed out, a much less invasive approach would have been to simply pause the server and get in touch with the client -- especially after the service provider asked for permission to get root access, and was turned down.
yep, I'd agree entirely with that :)
In this case their posted terms and conditions aren't very clear on the extent that they can muck around with a customer's VM. It's quite clear that they can suspend or kill it, but it's unclear how far they can go to "restrict" the customer's VM.
Rimuhostinh owns the box. The client rents it, but the owner never gives up the right to step in and fix something if it is impacting the service they provide for other clients.
If I require this level of privacy, I will go with a co-location contract, not a virtual machine. Those are different products with different features. A VM on a shared chassis is hardly a private computer.
Oddly, I have some experience with virtual servers and never had much trouble throttling down an instance that wanted too much resources.
If I require this level of privacy, I will go with a co-location contract, not a virtual machine. Those are different products with different features. A VM on a shared chassis is hardly a private computer.
Oddly, I have some experience with virtual servers and never had much trouble throttling down an instance that wanted too much resources.
Something similar happened to us a few days ago on our RimuHosting VPS. A Python script was eating ~80% of the VPS CPU, so their tech guy, without asking us, tries to login as root to our box (WTF?). He fails, so he responds by rebooting our VPS, thus (?) causing some disk corruption, which he then tries to repair. Then, after-the-fact, he writes us a very confused email about the whole story.
Needless to say, we're moving away from RimuHosting. This is unacceptable.
An excerpt:
"I noticed your VPS was using a lot of CPU. I was not able to log in there, and on the console I saw lots of out-of-memory messages.
I restarted the VPS to make sure it was running in a sane state. I checked the console after a reboot and saw an error indicating some filesystem corruption. I stopepd it again and repaired that. Then replaced the /etc/inittab file which appeared to be corrupted. Now I see that has booted fine."
Needless to say, we're moving away from RimuHosting. This is unacceptable.
An excerpt:
"I noticed your VPS was using a lot of CPU. I was not able to log in there, and on the console I saw lots of out-of-memory messages.
I restarted the VPS to make sure it was running in a sane state. I checked the console after a reboot and saw an error indicating some filesystem corruption. I stopepd it again and repaired that. Then replaced the /etc/inittab file which appeared to be corrupted. Now I see that has booted fine."
This article reminds me of Hard Disks. Talk to any "PC fan" and they'll regale you with a story of how they will never, ever use $HD_BRAND after it crashed and lost all their data. Thing is, the brand changes depending on who you talk to. And a real professional will tell you that all the brands have similar failure rates, not to put your faith in any hardware, and advise you on a RAID and backup strategy.
Same here. I know people who use rimuhosting, they seem happy enough. I use Linode personally but have no real preference. This article smacks of a one-off bad experience which could happen anywhere - and doesn't even seem that bad. It's a VPS, for christ's sake, if you don't like it get a dedi or even better a colo.
Same here. I know people who use rimuhosting, they seem happy enough. I use Linode personally but have no real preference. This article smacks of a one-off bad experience which could happen anywhere - and doesn't even seem that bad. It's a VPS, for christ's sake, if you don't like it get a dedi or even better a colo.
Shit, I hear the same stories about shipping companies. And cell phone companies. I now know never to use FedEx, UPS, or DHL.
You got it. Also, never visit America, Mexico, Canada, Britain, or any other country on earth.
Actually . . . you shouldn't visit the US. I'm leery of leaving the US because of the body cavity searches I can expect when I come back.
I left the U.S. for 2 years, visited shady places, had extra pages added to my passport after clocking 80 stamps on it to 22 countries. Was anxious to be harassed by the fat, ex-soldier redneck at the immigration desk. He took my passport, scanned it, and told me "welcome back home, son".
Congratulations. You had a good experience.
The reason I don't use consumer-grade Western Digital drives has nothing to do with a specific drive failure event, and everything to do with the fact that consumer-grade drives from WD are basically factory seconds of WD's enterprise-grade drives.
Which is exactly the kind of story that gets perpetuated over and over again, with different $brand's substituted. It doesn't even make sense: what criterion is used to seperate 'enterprise' from 'consumer'? They can't possibly test every drive (too expensive) and if they watch production parameters and decide based on those, chances are that most consumer drives are 'enterprise' as well. I find stories like these extremely implausible.
A nontrivial percentage of the "consumer" grade drives (basically, the stuff you can get at Best Buy) were actually manufactured as higher-capacity drives for the premium line (I forget the model line term WD uses at the moment) but had bad sectors coming off the line. They marked off those sectors, rounded down, low-level formatted so it only reported the "new" size, and sold them as lower capacity drives for retail consumer sales. Is that a more helpfully precise explanation?
That explanation still leaves some questions open, like: is there a reason to suppose that a drive with 'bad sectors coming off the line' actually has a higher expected failure rate? Or is it actually more like what happens with CPU's, where a batch may 'fail', but will still yield CPU that can be sold as 'lower' CPU's than the batch was originally supposed to deliver.
Marking something 'enterprise' grade is often a placebo and has more to do with service, contracts and well-marketed expectations than with actual differences in the goods involved.
Marking something 'enterprise' grade is often a placebo and has more to do with service, contracts and well-marketed expectations than with actual differences in the goods involved.
> is there a reason to suppose that a drive with 'bad sectors coming off the line' actually has a higher expected failure rate?
If it doesn't hurt me to choose a different brand or a different model line, the more relevant question becomes "Is there sufficient reason to believe that a drive with 'bad sectors coming off the line' doesn't have a higher actual failure rate?" It's better to be safe than sorry, as they say, and my experience is that once a drive starts degrading, it keeps degrading.
The reason CPUs can often be sold at a lower capacity than originally intended without worrying about an increased likelihood of later failure is because transistors are discrete devices; if half of them fail, as long as the rest can still be accessed, you just have a CPU with the same architecture and half the transistors. On a hard drive, however, permanently hosed sectors represent actual problems on part of a single discrete device -- the hard drive platter -- and it's entirely possible that the rest of the device may be affected by this. Magnetic areas on a platter are quite so distinctly segregated as transistors on a chip.
> Marking something 'enterprise' grade is often a placebo and has more to do with service, contracts and well-marketed expectations than with actual differences in the goods involved.
True -- but when you explicitly use drives that have failed the quality control required for "enterprise" drives as your retail consumer grade drives, that kinda changes the landscape a bit.
If it doesn't hurt me to choose a different brand or a different model line, the more relevant question becomes "Is there sufficient reason to believe that a drive with 'bad sectors coming off the line' doesn't have a higher actual failure rate?" It's better to be safe than sorry, as they say, and my experience is that once a drive starts degrading, it keeps degrading.
The reason CPUs can often be sold at a lower capacity than originally intended without worrying about an increased likelihood of later failure is because transistors are discrete devices; if half of them fail, as long as the rest can still be accessed, you just have a CPU with the same architecture and half the transistors. On a hard drive, however, permanently hosed sectors represent actual problems on part of a single discrete device -- the hard drive platter -- and it's entirely possible that the rest of the device may be affected by this. Magnetic areas on a platter are quite so distinctly segregated as transistors on a chip.
> Marking something 'enterprise' grade is often a placebo and has more to do with service, contracts and well-marketed expectations than with actual differences in the goods involved.
True -- but when you explicitly use drives that have failed the quality control required for "enterprise" drives as your retail consumer grade drives, that kinda changes the landscape a bit.
In my 2+ years hosting typeracer.com with Rimu, they have been absolutely outstanding. All my questions, no matter how complex, were answered within 10-30 minutes. Hundreds of times. Anything I've asked them they did. They installed all kinds of packages and performed configurations I requested, all for free and without hesitation. And I'm not a premium customer by any stretch. I only have 2 quarter-VPS instances with them for about $200 a month.
In the 2 years I only experienced 20 minutes of downtime recently due to a one-time network hardware upgrade (1 week's advance notice was given - a bit short but forgivable) and another 20 minutes due to a DDoS on their network about 6 months ago (which most likely prompted this recent hardware upgrade :).
I really hope people don't jump to conclusions from a single data point (aaronsw). I would recommend RimuHosting to everyone I know without hesitation.
In the 2 years I only experienced 20 minutes of downtime recently due to a one-time network hardware upgrade (1 week's advance notice was given - a bit short but forgivable) and another 20 minutes due to a DDoS on their network about 6 months ago (which most likely prompted this recent hardware upgrade :).
I really hope people don't jump to conclusions from a single data point (aaronsw). I would recommend RimuHosting to everyone I know without hesitation.
I'm with Rimuhosting since forever.
Their support is unbeatable. Many times they helped me(I'm a dev. not a sys. admin.) for free where other companies would have charged me.
I agree that adding their key on the VPS without talking to you sucks but having a 100% utilisation of your CPU is a bit egoist. That means that nobody on the host machine can burst their share for a quick high-cpu job.
If you want total control over your machine, get a dedicated box. Even then, some hosters demand root access on your box.
Their support is unbeatable. Many times they helped me(I'm a dev. not a sys. admin.) for free where other companies would have charged me.
I agree that adding their key on the VPS without talking to you sucks but having a 100% utilisation of your CPU is a bit egoist. That means that nobody on the host machine can burst their share for a quick high-cpu job.
If you want total control over your machine, get a dedicated box. Even then, some hosters demand root access on your box.
but having a 100% utilisation of your CPU is a bit egoist
I thought selling access to a shared resource without the ability to moderate consumption was the ridiculous bit.
I thought selling access to a shared resource without the ability to moderate consumption was the ridiculous bit.
nobody on the host machine can burst their share for a quick high-cpu job
That only makes sense if there is only one other VM on the machine. Which is basically never the case.
EDIT: Or they use a pedestrian virtualization layer.
That only makes sense if there is only one other VM on the machine. Which is basically never the case.
EDIT: Or they use a pedestrian virtualization layer.
I'm not sure I understand.
If there are 8 VPS on a machine and I'm always using 1/8 of CPU time, nobody else can't have access to this 1/8 share. I know that 1/8 of the CPU share "belongs" to me but it's a shared box and it's just being a good neighbour. I appreciate it when I need to run a quick high-cpu on my VPS and I can use more than my 1/8 share of CPU.
If there are 8 VPS on a machine and I'm always using 1/8 of CPU time, nobody else can't have access to this 1/8 share. I know that 1/8 of the CPU share "belongs" to me but it's a shared box and it's just being a good neighbour. I appreciate it when I need to run a quick high-cpu on my VPS and I can use more than my 1/8 share of CPU.
You'd need seven "bad" neighbors for this to be the case.
You're right but in this case the user is a freeloader.
He can use more than his 1/8 share of CPU time but others can't use his 1/8 share.
He can use more than his 1/8 share of CPU time but others can't use his 1/8 share.
Who cares? He should only be using the other 7/8s when nobody else is; that's the entire point. If their virtualization isn't setup to keep people from bogarting resources on a "first come, first keep" basis, that is their poor decision to live with; they shouldn't be taking it out on customers by killing processes and so on.
A VPS host shouldn't be asking or telling customers to manage their own CPU resources, period; that's the point of a VPS in the first place. It's a virtual private server. It should be isolated in a fashion such that even if I'm using 100% CPU 24/7, it doesn't affect anybody else. If a host told me to "use less CPU" I'd tell them to "acquire better virtualization" and find a better host. Ridiculous.
A VPS host shouldn't be asking or telling customers to manage their own CPU resources, period; that's the point of a VPS in the first place. It's a virtual private server. It should be isolated in a fashion such that even if I'm using 100% CPU 24/7, it doesn't affect anybody else. If a host told me to "use less CPU" I'd tell them to "acquire better virtualization" and find a better host. Ridiculous.
I agree but you should not be using a VPS if you are a high CPU user. You use a dedicated server for that.
Most shared providers will charge you more if you use more CPU or require that you upgrade to a dedicated server. I'm pretty sure that any hosting company will be happy to lose you as a customer if your usage degrades the other users performance.
Most shared providers will charge you more if you use more CPU or require that you upgrade to a dedicated server. I'm pretty sure that any hosting company will be happy to lose you as a customer if your usage degrades the other users performance.
Being able to use up to your share of a VPS is why you pay for a VPS and not shared hosting. If you pay for 1/8th of a server you are entitled to use 100% of that quota 24/7/365. Using your resources doesn't "degrade" other customers' performance unless the server is set up incorrectly. As long as they can access their 1/8th everyone should be happy.
> If there are 8 VPS on a machine
There's probably 64 or 128 VPS on one machine.
> and I'm always using 1/8 of CPU time
There's your problem right there. You're believing what you're told by someone selling shared access to a finite resource.
There's probably 64 or 128 VPS on one machine.
> and I'm always using 1/8 of CPU time
There's your problem right there. You're believing what you're told by someone selling shared access to a finite resource.
My perception is that VPS aren't for high-CPU, high-IO or high-memory servers. If one user is using too much CPU, he should be using a dedicated server. It's not like servers are expensive. You can get an entry level box on a fat pipe for 67$ with iweb.ca.
I think rimu was right to limit his CPU share. It was unprofessional of them to root his box and shutdown apache, but it seems he was a user with an history of high CPU utilization.
I think rimu was right to limit his CPU share. It was unprofessional of them to root his box and shutdown apache, but it seems he was a user with an history of high CPU utilization.
Wouldn't it have just made more sense to just further limit his CPU share? Even if he was abusing the service, I don't feel their actions were justified. Especially since they state in their email to him that if they cap the CPU then they don't care how much of his CPU share that he uses.
But Rimuhosting states in their email to him that they can give him a fixed CPU share, and then they don't care what he does with that. And they already had given him a fixed CPU share. If it was too large they could have further limited the size of his share, and then called him to notify him.
They ... offered to take a look at the problem if I gave them root on the box. “Over my dead body,” I thought
That seems a little silly to me. If they control the hardware, they basically have root already. They shouldn't need root to investigate the alleged problem, of course, but the level of trust you need to grant your VPS provider is essentially the same as giving them root.
That seems a little silly to me. If they control the hardware, they basically have root already. They shouldn't need root to investigate the alleged problem, of course, but the level of trust you need to grant your VPS provider is essentially the same as giving them root.
Technically the police can break down my door too, but that doesn't mean that I'm going to trust them with a key. There are both social reasons (a key is an invitation) and technical (it's much more noticeable if they break down the door).
There's a social difference, yes. In this case, I'm not sure the technical difference is all that large: if they were at all competent, they could have easily snooped on your VM without your knowledge.
Yes, but in this case they weren't breaking down your door (at least not yet anyway). They were asking for a key for entry in order to assist you.
You could have most likely also given them an account and appropriate sudoer rights.
You could have most likely also given them an account and appropriate sudoer rights.
Did you read the article? They edited my partition to add their key to my authorized_keys, logged in with it, and turned off my webserver. Whatever that is, it's not asking to assist me.
I did read the article. Based on what you wrote, they offered to help investigate the issue in October, November, December if you would give them root access. For some reason you had an issue with this (it's actually extremely common on any shared box that you do not own).
After the server that you lease from them (I omit calling it "your" server on purpose) continued to have issues that impacted customers outside of yourself, they logged into the box and (based on your article again) seemed to resolve the root (pun intended) issue rather quickly. This seems to have occurred 2-3 months after the initial root access request.
The response of shutting off Apache was a bit extreme, but given the history, not terribly surprising either.
After the server that you lease from them (I omit calling it "your" server on purpose) continued to have issues that impacted customers outside of yourself, they logged into the box and (based on your article again) seemed to resolve the root (pun intended) issue rather quickly. This seems to have occurred 2-3 months after the initial root access request.
The response of shutting off Apache was a bit extreme, but given the history, not terribly surprising either.
If you rent an apartment, you don't call it "the apartment that I rent", you call it "my apartment". Also, the landlord owns the place, but that doesn't give the landlord carte blanche to enter the apartment anytime they feel like it. Sure, they "own" it, but that does not make it any less wrong for them to barge in and gain access. If they need access, they must provide sufficient notice and must do it for a specific reason. This is a very similar situation.
Particularly interesting to me is that they had the technical means to restrict the virtual machine via a configuration change, but instead chose to "trespass".
According to their terms and conditions, "Misuse of System Resources" is prohibited, but their only listed recourse is to "restrict, suspend, or terminate" the account. It is arguable that they had no right to access the data for his server, since they had other means to restrict or otherwise limit the VPS (which they already did). They should have just paused the VPS without accessing the data and send him an email. Adding your SSH key to a customer's server without notice or approval is a good way to tarnish a reputation among the hacker crowd (their customer base). I wouldn't allow them to have root access either.
Their argument would have been much stronger had they provided any sort of evidence of CPU load of his server. However, that probably would have shown how oversold the machine really was.
Particularly interesting to me is that they had the technical means to restrict the virtual machine via a configuration change, but instead chose to "trespass".
According to their terms and conditions, "Misuse of System Resources" is prohibited, but their only listed recourse is to "restrict, suspend, or terminate" the account. It is arguable that they had no right to access the data for his server, since they had other means to restrict or otherwise limit the VPS (which they already did). They should have just paused the VPS without accessing the data and send him an email. Adding your SSH key to a customer's server without notice or approval is a good way to tarnish a reputation among the hacker crowd (their customer base). I wouldn't allow them to have root access either.
Their argument would have been much stronger had they provided any sort of evidence of CPU load of his server. However, that probably would have shown how oversold the machine really was.
If you rent an apartment, you don't call it "the apartment that I rent", you call it "my apartment". Also, the landlord owns the place, but that doesn't give the landlord carte blanche to enter the apartment anytime they feel like it.
This analogy doesn't really hold. "Your" apartment gets special rights under the law in the US and probably other places. Those rights are actually pretty extreme (it takes months after they stop paying to evict someone in most states, for example), and you don't have those same rights or expectations of privacy within an office building, or anything else you rent or lease that isn't your domicile.
According to their terms and conditions, "Misuse of System Resources" is prohibited, but their only listed recourse is to "restrict, suspend, or terminate" the account.
Sounds, to me, like they restricted his account to one that wasn't running Apache.
But, I agree that they should have simply turned him off. Customers that don't want administrative assistance on the system should never have it imposed on them, and I would also prefer my provider never login to my systems (and I don't provide them with access to my systems; though I know that because they have access to the hardware, they always have root available to them; I rely on good faith to prevent them from doing harm or abusing my privacy).
However, that probably would have shown how oversold the machine really was.
While this is very often true in the VPS hosting industry, Rimuhosting has an otherwise very good reputation. It's pretty easy to know when you're on an oversold host, and it's not something I've ever heard Rimuhosting accused of.
This analogy doesn't really hold. "Your" apartment gets special rights under the law in the US and probably other places. Those rights are actually pretty extreme (it takes months after they stop paying to evict someone in most states, for example), and you don't have those same rights or expectations of privacy within an office building, or anything else you rent or lease that isn't your domicile.
According to their terms and conditions, "Misuse of System Resources" is prohibited, but their only listed recourse is to "restrict, suspend, or terminate" the account.
Sounds, to me, like they restricted his account to one that wasn't running Apache.
But, I agree that they should have simply turned him off. Customers that don't want administrative assistance on the system should never have it imposed on them, and I would also prefer my provider never login to my systems (and I don't provide them with access to my systems; though I know that because they have access to the hardware, they always have root available to them; I rely on good faith to prevent them from doing harm or abusing my privacy).
However, that probably would have shown how oversold the machine really was.
While this is very often true in the VPS hosting industry, Rimuhosting has an otherwise very good reputation. It's pretty easy to know when you're on an oversold host, and it's not something I've ever heard Rimuhosting accused of.
The apartment analogy isn't perfect, but it is a good example to show that just because you own something, it doesn't give you the legal (or moral) right to muck around with it whenever you please. If you're renting computer space to a customer, there are rules that need to be followed. Some of them are legal, some contractual, and some are societal norms.
In this case they definitely violated the societal norms and possibly their contractual obligations as well. I'd say that you have a pretty good expectation of privacy on a virtual _private_ server.
I don't particularly know anything about Rimuhosting. I know most places are oversold to a certain degree. I suspect that there isn't any way to make money in the industry if you're not.
In this case they definitely violated the societal norms and possibly their contractual obligations as well. I'd say that you have a pretty good expectation of privacy on a virtual _private_ server.
I don't particularly know anything about Rimuhosting. I know most places are oversold to a certain degree. I suspect that there isn't any way to make money in the industry if you're not.
* Also, the landlord owns the place, but that doesn't give the landlord carte blanche to enter the apartment anytime they feel like it.*
Of course not, but the landlord does generally have the right to enter the property if there is an emergency (water is dripping from your unit into the unit below you).
The hosting company here did not just drop in to see if he had any good scripts or pr0n to steal, they did so after resource issues.
Of course not, but the landlord does generally have the right to enter the property if there is an emergency (water is dripping from your unit into the unit below you).
The hosting company here did not just drop in to see if he had any good scripts or pr0n to steal, they did so after resource issues.
The apartment manager wouldn't have the ability to "pause" the apartment and wait for permission from the tenant to enter, though. The situation isn't strictly analogous.
Exactly. In my mind this is the difference. If there is an emergency, the hypothetical apartment manager must physically enter the apartment.
However, in this case, the hosting company has another option available: pause the VM.
It is also easier to do technically than shutdown the VM, edit the filesystem, restart the VM, find the errant process, change the configuration, etc....
However, in this case, the hosting company has another option available: pause the VM.
It is also easier to do technically than shutdown the VM, edit the filesystem, restart the VM, find the errant process, change the configuration, etc....
How is pausing an entire slice and thereby killing ALL services better than just shutting down one daemon?
It's better the same way that your apartment complex manager turning off the water to your apartment if there's a problem is better than going inside when you explicitly told him not to, checking all your faucets, and going through your underwear drawer, when you're not there.
Rimuhosting didn't go through this guy's underwear drawer, though. They went in and turned off the faucet that was damaging the apartment below.
> Rimuhosting didn't go through this guy's underwear drawer, though.
How do you know? They rooted the system against the explicitly stated wishes of the customer. What makes you think they wouldn't do other sketchy things while they're at it?
How do you know? They rooted the system against the explicitly stated wishes of the customer. What makes you think they wouldn't do other sketchy things while they're at it?
What if there were multiple interconnected daemons running? What if shutting down apache stopped the apache process from doing some other job that another process needed? With servers that you don't maintain, you just don't always know how the services interact.
So, it is always better to just pause the entire VM and let the owner deal with it. (Sure, you can offer to help fix it, that's good customer service, but just don't take it upon yourself to do it without approval).
So, it is always better to just pause the entire VM and let the owner deal with it. (Sure, you can offer to help fix it, that's good customer service, but just don't take it upon yourself to do it without approval).
You could have used a tool I wrote http://shellshadow.com
Then you could have watched them investigate the problem.
Then you could have watched them investigate the problem.
That looks handy. you can use screen to do a similar thing which is much easier to install on the actual Linux server, and works with windows, Linux and mac
One person starts a screen session, and the other person uses screen -x to share it.
Screen is a good tool and does have a collaborative feature. But it requires the following:
1 - install screen and configure the collaboration on your server
2 - create a user on your server for the other person to login
3 - the other person logging in must be able to route directly to your server. The ip for the server must be public and SSHD listener must accept connect from anywhere.
4 - The other person can login and "collaborate" with your screen session whether you are logged-in/watching or not. The only security is to shut down your screen session or change your screen config file / stop the sharing.
5 - with screen you cannot interactively/quickly change modes between read-write, read-only, etc...
ShellShadow works as a client terminal relay. You setup a session between you and another user. You use the ShellShadow client (enhanced PuTTY) for your collaboration. There is nothing to setup and manage on your server. In fact your server is oblivious to the collaboration. One extra cool feature is this works with any terminal you can connect to: your Cisco router, your iPhone CLI, etc...
1 - install screen and configure the collaboration on your server
2 - create a user on your server for the other person to login
3 - the other person logging in must be able to route directly to your server. The ip for the server must be public and SSHD listener must accept connect from anywhere.
4 - The other person can login and "collaborate" with your screen session whether you are logged-in/watching or not. The only security is to shut down your screen session or change your screen config file / stop the sharing.
5 - with screen you cannot interactively/quickly change modes between read-write, read-only, etc...
ShellShadow works as a client terminal relay. You setup a session between you and another user. You use the ShellShadow client (enhanced PuTTY) for your collaboration. There is nothing to setup and manage on your server. In fact your server is oblivious to the collaboration. One extra cool feature is this works with any terminal you can connect to: your Cisco router, your iPhone CLI, etc...
You make it sound more difficult than it is. In other words, they mounted your partition, cat mykey.txt > authorized_keys, then ssh'd in and shut down your misbehaving app. They probably have a script for it since I bet it happens all the time.
You're dangerously close to whining, did you know.
You're dangerously close to whining, did you know.
How hard it was for them to do something isn't the issue.
From the blog post:
> In December they set a CPU cap on my VPS.
Then from their response email to shutting down Apache:
> We can set it so that you get a fixed amount of CPU (and then we don’t mind how much CPU want to use).
My first reaction is, "Huh?" They state that they don't care about your local CPU usage if they bother to put a cap on your usage of the actual CPU. Yet they didn't do that. Instead they broke into his box to shutdown a rogue process that was 'using too much cpu.' That doesn't make any sense.
From the blog post:
> In December they set a CPU cap on my VPS.
Then from their response email to shutting down Apache:
> We can set it so that you get a fixed amount of CPU (and then we don’t mind how much CPU want to use).
My first reaction is, "Huh?" They state that they don't care about your local CPU usage if they bother to put a cap on your usage of the actual CPU. Yet they didn't do that. Instead they broke into his box to shutdown a rogue process that was 'using too much cpu.' That doesn't make any sense.
You make it sound more difficult than it is.
Hmm. I'm sorry, but I don't see how this matters at all Sho. Do the ethics change based on whether you have a rootkit or you do everything by hand?
Hmm. I'm sorry, but I don't see how this matters at all Sho. Do the ethics change based on whether you have a rootkit or you do everything by hand?
Ethics? Ethics don't enter into it, this is pure business necessity. Fact is, a $20/month VPS account has no rights and if it causes trouble it will be shut down, simple as that.
Pay $250/month for an account at Rackspace (or whoever) and you will be treated very differently.
Pay $250/month for an account at Rackspace (or whoever) and you will be treated very differently.
Pay $250/month for an account at Rackspace (or whoever) and you will be treated very differently.
Actually, the reason you pay $250 (more, probably) at Rackspace, is because they will login to your box on a regular basis to "manage" things. Rackspace is focused on "managed hosting"; they help you administer your system, which is why people pay a big premium for it. Most hosting customers have no idea what they're doing, and they need a lot of hand-holding, and a "grownup" to make sure things stay sane on their systems.
Actually, the reason you pay $250 (more, probably) at Rackspace, is because they will login to your box on a regular basis to "manage" things. Rackspace is focused on "managed hosting"; they help you administer your system, which is why people pay a big premium for it. Most hosting customers have no idea what they're doing, and they need a lot of hand-holding, and a "grownup" to make sure things stay sane on their systems.
True. A rackspace account is more like $450 for a single server. They keep their prices secret for a reason.
It would be easy as hell for me to shoot the hinges off a neighbor's door with a shotgun, wander inside, and turn off all the lights. That doesn't make it right.
I think it isn't. Even if they have physical access to the machine, breaking into it is ethically questionnable and highly unprofessional. Unless they detect some sort of illegal activity or there is a real emergency, this is just simply unacceptable.
From their point of view, he'd repeatedly ignored three separate requests to clean up his CPU usage. After he refused to do anything about it, they took care of the problem. Would it have been better if they disabled his account entirely?
But they had already capped his CPU usage. I see only 3 possibilities:
* Their cap failed to actually cap his CPU usage. In this case their incompetence is at fault.
* They failed to make the cap restrictive enough. In which case, they should have just further limited his max actual CPU usage.
* The box is actually well oversold on the number of clients that are on it, and there is no possible way to 'cap' someone's CPU usage to within a reasonable level while being able to allow that person to use 100% of their cap without issue. In this case, it's also their fault for over-selling the box and for misrepresenting what a cap means (and their policy regarding caps).
* Their cap failed to actually cap his CPU usage. In this case their incompetence is at fault.
* They failed to make the cap restrictive enough. In which case, they should have just further limited his max actual CPU usage.
* The box is actually well oversold on the number of clients that are on it, and there is no possible way to 'cap' someone's CPU usage to within a reasonable level while being able to allow that person to use 100% of their cap without issue. In this case, it's also their fault for over-selling the box and for misrepresenting what a cap means (and their policy regarding caps).
I have been using RimuHosting for my own projects and for customers for test deployments for years - so far I have had good experiences with them. They did complain once that I was using too much CPU, but I checked, and found problems (my fault) with a Merb deployment).
In a way, isn't this a reason to use Rimuhosting. They obviously care about their customers enough, to proactively monitor and step in that everyone gets a fair CPU share.
You can make sure that people get a fair CPU share without resorting to such primitive (and wrong) methods.
Xen has the ability to cap CPU usage with either hard or soft caps. Most VPS companies seem to use soft caps. That way, if there are 4 instances on the box and each is doing lots of stuff, they'll each get 25%. But if 3 of the instances are mostly idle not using CPU, but one needs more CPU it will be able to use extra. It works out really nicely because it means that your usage can burst at certain times when it's unlikely that your co-habitants are similarly bursting.
See: http://www.slicehost.com/questions/#cpu-scheduling
Xen has the ability to cap CPU usage with either hard or soft caps. Most VPS companies seem to use soft caps. That way, if there are 4 instances on the box and each is doing lots of stuff, they'll each get 25%. But if 3 of the instances are mostly idle not using CPU, but one needs more CPU it will be able to use extra. It works out really nicely because it means that your usage can burst at certain times when it's unlikely that your co-habitants are similarly bursting.
See: http://www.slicehost.com/questions/#cpu-scheduling
They showed their lack of technical proficiency. CPU caps should be enforced without having to edit partitions, adding ssh keys to users accounts, asking for root passwords, or manually killing Apache.
So to put it bluntly they are very nice and helpful, but stupid. I wouldn't trust them with my data. But I can see how some customers might prefer them, as there is always a nice person on the other end of the line...
So to put it bluntly they are very nice and helpful, but stupid. I wouldn't trust them with my data. But I can see how some customers might prefer them, as there is always a nice person on the other end of the line...
If you were to believe the author, his VPS wasn't causing high CPU usage.
Regardless, doesn't most vps instances have limitations built into their VPS instances? It shouldn't matter to other VPS in that box how much CPU is being used by other VPS, as long as they are within their limitations?
That was my experience with slicehost and other VPS
Regardless, doesn't most vps instances have limitations built into their VPS instances? It shouldn't matter to other VPS in that box how much CPU is being used by other VPS, as long as they are within their limitations?
That was my experience with slicehost and other VPS
Except most providers grossly oversell their VPS instances just like bandwidth, and when someone actually starts using 100% of their quoted capacity, it throws everything out of equilibrium.
From bizarre reactions to imaginary CPU use problems, to hacking in to their client's servers, this article presented some good reasons not to use Rimuhosting.
That's sorta like a hotel renting a room... and if you overstay your rent, they throw you out the window. Sure it's their room, they're allowed to do anything they want with it, but I wouldn't want to be that guy thrown out the window either. There are better ways to solve the problem than using a shotgun.
Except that the need to manually deal with CPU usage shows that they are amateur idiots.
There's no reason to be assigning VPSes to particular machines in 2009 -- distribute the damn things instead of selling shared hosting like in the bad old days (except this time with VM overhead!).
There's no reason to be assigning VPSes to particular machines in 2009 -- distribute the damn things instead of selling shared hosting like in the bad old days (except this time with VM overhead!).
Doesn't Xen do proper CPU sharing (but not proper I/O sharing, yet)? What are they using?
Would you mind posting what the offending CGI was doing? If you coded up your index.cgi to walk all over the CPU as badly as it sounds like you did, something was probably very very wrong.
That said, they were absolutely in the right to do this (except that they should have notified you first...).
I saw that you (or somebody) used the analogy of a landlord and an apartment. Let's explore that further...
Say you are renting an apartment, and the landlord doesn't charge you anything for water (which is pretty common, if not standard). Now, there is a bit of a gentlemen's agreement here. Yes, you have "unlimited" water but, like everything else in life, this needs to have the words "within reason" appended to it.
Now lets say that, for some reason, your apartment was using 10x as much water as everybody else in the building...so much so, in fact, that nobody else in the building could reliably get any water.
Your landlord calls you and says "Hey, joao, would you mind cutting the water back, please? You're kindof using too much", which you ignore.
A couple of months later, again, the landlord calls
"Hey, Joao, you're still using a shit-load of water, and your sortof hogging it all. Some of the other tenants are complaining...if you want, leave a key to your place in the office and we'll go check it out!"
Which you respond "over my dead body".
So, finally, after months and months of you hogging the water, they just use the master key to the building to go into your apartment, where they see that you've had the shower, the sink, and the toilet all running constantly for the last 4 months. They shut them off, then call you
"Hey, joao, yeah, we noticed that you pretty royally messed up and kindof left EVERYTHING in the bathroom on at the same time. Not cool...so we shut it off. Sorry for the inconveinience. BTW, sorry for having to go into your appt. like that, but the other tenants were complaining. Here are some instructions so that you can change the lock on your door an keep us from being able to help you like this in the future. Have a nice day!"
And your response to ignoring their totally legitimate requests for you to stop hogging all the resources is "WHAT A BUNCH OF PRICKS!!"
I think somebody needs to calm down and take a step back from the situation.
That said, they were absolutely in the right to do this (except that they should have notified you first...).
I saw that you (or somebody) used the analogy of a landlord and an apartment. Let's explore that further...
Say you are renting an apartment, and the landlord doesn't charge you anything for water (which is pretty common, if not standard). Now, there is a bit of a gentlemen's agreement here. Yes, you have "unlimited" water but, like everything else in life, this needs to have the words "within reason" appended to it.
Now lets say that, for some reason, your apartment was using 10x as much water as everybody else in the building...so much so, in fact, that nobody else in the building could reliably get any water.
Your landlord calls you and says "Hey, joao, would you mind cutting the water back, please? You're kindof using too much", which you ignore.
A couple of months later, again, the landlord calls
"Hey, Joao, you're still using a shit-load of water, and your sortof hogging it all. Some of the other tenants are complaining...if you want, leave a key to your place in the office and we'll go check it out!"
Which you respond "over my dead body".
So, finally, after months and months of you hogging the water, they just use the master key to the building to go into your apartment, where they see that you've had the shower, the sink, and the toilet all running constantly for the last 4 months. They shut them off, then call you
"Hey, joao, yeah, we noticed that you pretty royally messed up and kindof left EVERYTHING in the bathroom on at the same time. Not cool...so we shut it off. Sorry for the inconveinience. BTW, sorry for having to go into your appt. like that, but the other tenants were complaining. Here are some instructions so that you can change the lock on your door an keep us from being able to help you like this in the future. Have a nice day!"
And your response to ignoring their totally legitimate requests for you to stop hogging all the resources is "WHAT A BUNCH OF PRICKS!!"
I think somebody needs to calm down and take a step back from the situation.
Great analogy, but you missed the point. (Great analogy because I can still use it to demonstrate the point) I would be pretty pissed if my landlord use the master key to access my apartment when he had a valve in the utility closet to just shut-off all water to my apartment.
These are VPSes here. They can just pause the entire VPS instance, or further cap the CPU usage from what they had already capped it at.
These are VPSes here. They can just pause the entire VPS instance, or further cap the CPU usage from what they had already capped it at.
Hooray for this analogy! Sadly, you didn't really use it correctly :(.
Pausing the VPS instance would have been like locking them out of the apartment completely...
I would much rather have a landlord come into my apartment to turn my faucet off (keep in mind: after I ignored her requests to for MONTHS) than lock me out of it completely.
Pausing the VPS instance would have been like locking them out of the apartment completely...
I would much rather have a landlord come into my apartment to turn my faucet off (keep in mind: after I ignored her requests to for MONTHS) than lock me out of it completely.
You missed the 'or further limit his CPU share' part. The entire point was that they had options to limit his affect on other users without directly accessing his instance.
In this case, they also have given the users the option of the 'master key.' Their ssh key is in the authorized_keys file by default, but the users are allowed to removed it. To me that is a statement to Rimuhosting that this customer would prefer that they did not access his VPS directly.
In this case, they also have given the users the option of the 'master key.' Their ssh key is in the authorized_keys file by default, but the users are allowed to removed it. To me that is a statement to Rimuhosting that this customer would prefer that they did not access his VPS directly.
This is the only time I've seen anyone in this thread ask what the process was. I voted you up for that. Did the offending proccess really warrant the kind of intervention that RimuHosting is being criticized of?
Am I the only one thinking that Rimuhosting is a rather unfortunate name?
Fanatical service, man.
It's the name of a tree native to New Zealand (where the company is located): http://en.wikipedia.org/wiki/Dacrydium_cupressinum
Couldn't agree more. I got a VPS from them after asking HN for VPS recommendations in Europe.
Had issues from the start. The setup wasn't what I ordered. Once it was setup properly, I started getting alerts from their staff that my VPS's resources were spiking–but this was before I had even logged into the system and done anything. That didn't do much for my confidence in their operation.
Anyway, I canceled with them and will probably use Rackspace as they have UK/European data centers.
Had issues from the start. The setup wasn't what I ordered. Once it was setup properly, I started getting alerts from their staff that my VPS's resources were spiking–but this was before I had even logged into the system and done anything. That didn't do much for my confidence in their operation.
Anyway, I canceled with them and will probably use Rackspace as they have UK/European data centers.
I signed up with RimuHosting a few months ago, since everyone was raving about their competent and responsive support. So far it has been a mixed bag for me. They offered to set up some services for me when I was signing up (a monitoring service and a pop3 server). Whoever was setting these up did a horrible job; I'm talking about basic mistakes in the config files that prevented these services from even working. So I would not trust RimuHosting to set up anything on my server.
On the other hand, when I experienced weird routing problems between my machine and my home, Rimu eventually offered to move me to a more expensive London VPS at the same price, even though the dropped packets were not their fault. So they are willing to go out of their way to make a customer happy.
As I said, a mixed bag. I'm not planning on changing hosts anytime soon.
On the other hand, when I experienced weird routing problems between my machine and my home, Rimu eventually offered to move me to a more expensive London VPS at the same price, even though the dropped packets were not their fault. So they are willing to go out of their way to make a customer happy.
As I said, a mixed bag. I'm not planning on changing hosts anytime soon.
[deleted]
I've used Future Hosting http://www.futurehosting.com/ for a few years now, with two different VPS instances for $30/month each. Never had a problem, and if they did something like this I'd be shocked.
[deleted]
It can't be left unsaid: I assumed (no pun intended) this was a joke posting when I first saw it, seriously "Rim You Hosting"??!
I've had good service from Linode for nearly 2 years: http://www.linode.com/?r=73dc8f5748a14c821e64febc5e461a606f6...
was the referral link really necessary?
That went well for you.
But, I've had bad experiences with literally every single host I've ever used (dozens of them), and the longer you stay with a provider, the more likely you are to find a tech having a bad day, or their network fiber getting cut by the utility company, or various other demonstrations of man's fallibility.
I think the good probably still outweighs the bad with Rimuhosting...as Aaron mentioned, they do have a good reputation among many people who ought to know. The horror stories I hear about really bad hosts leave this event in the dust in terms of sheer stupidity and lack of good intentions. If this is the worst you've ever experienced with a hosting provider, count yourself very, very lucky. The data loss stories I've heard, in particular, are enough to keep me awake at night.
In the interest of full disclosure, I'll mention that Rimuhosting is a customer of ours (along with thousands of others). So, if you reckon a commercial relationship worth a few hundred bucks a year would sway my opinions, feel free to take this comment with a grain of salt or not at all.