Security Notice: Linode Manager Password Reset(blog.linode.com)
blog.linode.com
Security Notice: Linode Manager Password Reset
http://blog.linode.com/2013/04/12/security-notice-linode-manager-password-reset/
70 comments
Not all customers. I didn't receive any notice.
so every time someone tries to make unauthorized attempts to access a single customer's account.. all customers are to be required to reset their passwords?
That's the part that makes me wonder. The email that I received said, "We have found no evidence that any Linode data of any other customer was accessed."
I suspect that they aren't 100% sure, and that this is a precaution. I can live with it, but I hope that they keep their customers in the loop on things, where possible.
I suspect that they aren't 100% sure, and that this is a precaution. I can live with it, but I hope that they keep their customers in the loop on things, where possible.
> We have been advised that law enforcement officials are aware of the intrusion into this customer’s systems.
I feel like this is probably more than just some dude's account.
I feel like this is probably more than just some dude's account.
I think this as well. Why did Linode describe themselves as being the ones advised, rather than Linode being the one to inform law enforcement of the breach?
Right? I don't see how this helps.
It could be that they are restricted from providing more details due to the nature of the issue?
It sounds like a cop-out and is, but if for instance it was under active investigation, and it was something big, they might be able to reset customer passwords while not identifying information to the attacker.
On the other hand, this type of action would give it away either way, so maybe even that excuse doesnt smell right.
On the other hand, this type of action would give it away either way, so maybe even that excuse doesnt smell right.
Now the question is who got hacked?
If they take actions like this.
I thought this seemed familiar..
http://status.linode.com/2012/03/manager-security-incident.h...
http://status.linode.com/2012/03/manager-security-incident.h...
I wonder if this is another attempt at hacking an account containing bitcoins? Last time the thieves made out with close to 50k BTC if I recall correctly.
It's good that Linode is taking security seriously, but the pessimist in me wonders; if all it takes to get a password reset site-wide is an attack on a single user, wouldn't that open up a whole new, rather aggravating attack aimed solely at making users fed up with having their password reset all the time?
I'm not sure I'd describe Linode's security stance as 'serious.'
Last I knew, they were running a seriously deprecated OS version for their hosts, and their configuration management systems left a lot to be desired in terms of security.
Perhaps more worrying is how opaque they are about everything. Nevermind the security issues, they won't even provide explanations for 'normal' service outages.
Last I knew, they were running a seriously deprecated OS version for their hosts, and their configuration management systems left a lot to be desired in terms of security.
Perhaps more worrying is how opaque they are about everything. Nevermind the security issues, they won't even provide explanations for 'normal' service outages.
> It's good that Linode is taking security seriously
From what it seems, the only thing they take seriously is responding to incidents like these and letting customers know. If they were actually serious about security, these things wouldn't be happening. It was almost over a year ago since the last event.
From what it seems, the only thing they take seriously is responding to incidents like these and letting customers know. If they were actually serious about security, these things wouldn't be happening. It was almost over a year ago since the last event.
[deleted]
Welp, it looks like it's time to move to Amazon EC2. It's cheaper and hasn't (to my knowledge) been hacked yet.
Not sure why HN is full of so many fair-weather friends these days. Then again your account is only a few days old. I don't wanna sound like an old geezer telling you to get off my lawn, but you should really read the guidelines for posting on HN. For the sake of genuine discussion and just better social behavior in general. Could you say this out loud to a bunch of other hackers with a straight face?
What did I say wrong? I'm just stating my plans after hearing about another security blunder on Linode's part. Did what I say come off as sarcastic?
Can you elaborate on why you think it is cheaper? It's always appeared to me that you pay a hefty premium for compute resources on AWS (at least compared to Linode).
[deleted]
It's probably false that "there does not exist a customer of EC2 that has been hacked", and that should be pretty obvious. Furthermore it's false for essentially all nontrivial, large services.
But here, it's not a customer that was hacked. It was linode that was hacked, and used to access a specific customer's data.
"We have been advised that law enforcement officials are aware of the intrusion into this customer’s systems." sounds like a customer was hacked, not Linode.
If a customer was hacked, why reset everyone's password? Unless there is something Linode is not telling us, there is no reason they should be doing this. Think about it like this: what if Google reset everyone's passwords whenever a gmail account got compromised? Ridiculous.
Then why was my password reset?
> Out of an abundance of caution, however, we have decided to implement a Linode Manager password reset. In so doing, we have immediately expired all current passwords.
In other words, better safe than sorry.
In other words, better safe than sorry.
If the compromise was a brute force, then there's no need to reset my password.
If the compromise was due to a flaw in Linode's system (potentially exposing other accounts) then a global password reset makes sense.
Can you imagine if every service you used reset everyones passwords every time one of their users got brute forced? You'd do nothing but reset your passwords all day...
If the compromise was due to a flaw in Linode's system (potentially exposing other accounts) then a global password reset makes sense.
Can you imagine if every service you used reset everyones passwords every time one of their users got brute forced? You'd do nothing but reset your passwords all day...
[citation needed]
Amazon AWS does have much better security for their control panel/admin stuff than most random VPS providers. Linode is particularly bad as VPSes go.
irrelevant argument at best
I have said it before: I have a hard time trusting Linode after the major March 2012 security incident (HN thread: https://news.ycombinator.com/item?id=3654110 ; my comment: https://news.ycombinator.com/item?id=5339799)
At the time they did not give much information. They did not do a follow up. They did not discuss plans to prevent the same type of breach. I am not surprised that today, they got breached again! sigh
And again, they are making the same mistakes. They are not giving much information. They are not going to do a follow up. Etc.
At the time they did not give much information. They did not do a follow up. They did not discuss plans to prevent the same type of breach. I am not surprised that today, they got breached again! sigh
And again, they are making the same mistakes. They are not giving much information. They are not going to do a follow up. Etc.
As a former developer of Bitcoinica, the most affected customer in the previous Linode incident at that time, I have to say that Linode's communication was extremely unresponsive. It took them 15 days to write a reply that is still full of ambiguity.
In the end they offered Bitcoinica one-year worth of services, which are basically nothing compared to the huge losses. I can expect 7 other customers to be treated similarly.
There were no details available at all. Not only they didn't follow up to the public, there's a shockingly lack of transparency to their affected customers as well.
In the end they offered Bitcoinica one-year worth of services, which are basically nothing compared to the huge losses. I can expect 7 other customers to be treated similarly.
There were no details available at all. Not only they didn't follow up to the public, there's a shockingly lack of transparency to their affected customers as well.
In addition, the web interface lacks:
- 2-factor auth [1]
- X-Frame-Options [2]
- HSTS [2]
- Redirect from http to https prior to login. (if someone bookmarks the http url, for example, they can be mitm'd anytime they use that bookmark) [2]
These have been pointed out on the forums long ago (see cites) but have not been implemented.
[1] http://forum.linode.com/viewtopic.php?f=7&t=7639
[2] http://forum.linode.com/viewtopic.php?f=7&t=8764
- 2-factor auth [1]
- X-Frame-Options [2]
- HSTS [2]
- Redirect from http to https prior to login. (if someone bookmarks the http url, for example, they can be mitm'd anytime they use that bookmark) [2]
These have been pointed out on the forums long ago (see cites) but have not been implemented.
[1] http://forum.linode.com/viewtopic.php?f=7&t=7639
[2] http://forum.linode.com/viewtopic.php?f=7&t=8764
It doesn't always redirect...
If you visit http://manager.linode.com/session/index it'll happily let you log in via HTTP.
If you visit http://manager.linode.com/session/index it'll happily let you log in via HTTP.
Why do you think in both cases this was the result of some sort of mistake on Linode's part, versus an oversight on the part of the target?
That is, you use the expression 'they {linode} got breached', versus the more correct 'they {the customers} got breached'. How do you know that in both cases this was the case?
That is, you use the expression 'they {linode} got breached', versus the more correct 'they {the customers} got breached'. How do you know that in both cases this was the case?
The wording they use makes it sound like the attackers compromised some Linode service to gain access to a specific customers machine rather than that customer themselves being exploited. If it was just a compromise of that customer there would obviously be no need for this announcement or resetting of passwords. At the very least they don't know which it was.
In March 2012, Linode acknowledged it was a breach of their internal systems, not the customers'.
Today, they are reseting the password of all customers, making it pretty certain this is another breach at the Linode level, not at the customer's level.
Today, they are reseting the password of all customers, making it pretty certain this is another breach at the Linode level, not at the customer's level.
If it was one customer's fault, why reset all Linode users' passwords?
If you knew there was a cracker actively using your network, would you not take any other precautions other than changing the one password you know he used?
One could argue Linode was overzealous in resetting all passwords. Reasonable minds can differ.
One could argue Linode was overzealous in resetting all passwords. Reasonable minds can differ.
That depends completely on the method of compromise...
If the compromise was due to a weak password that was brute forced, or obtained from the end user in some other way, then there is absolutely no need to reset every user's password.
The only time a 'reasonable mind' would think to reset everyone's password is if the attacker exploited some flaw in the underlying system, which could have given them access to arbitrary passwords.
If the compromise was due to a weak password that was brute forced, or obtained from the end user in some other way, then there is absolutely no need to reset every user's password.
The only time a 'reasonable mind' would think to reset everyone's password is if the attacker exploited some flaw in the underlying system, which could have given them access to arbitrary passwords.
Yeah, it does sound like there's something to the story that they aren't disclosing, and it doesn't inspire confidence.
Malicious traffic doesn't make routers burst into flame. Customers get compromised on a daily basis at any large provider due to bad SSH passwords or WordPress vulnerabilities or whatever.
Assuming the virtualization system is trustworthy (which it is), that's not a problem. Assuming it's not trustworthy, they shouldn't be running a virtual hosting company and letting random strangers sign up in the first place!
Assuming the virtualization system is trustworthy (which it is), that's not a problem. Assuming it's not trustworthy, they shouldn't be running a virtual hosting company and letting random strangers sign up in the first place!
Any decently sized web host is almost certain to have crackers actively using their network at any time. AWS probably has thousands of instances running old WP installs that've been compromised.
The lack of transparency from them makes it impossible to know one way or the other.
The xen security issue is another example of this kind of opaqueness.
The xen security issue is another example of this kind of opaqueness.
It just occurred to me that https://www.schneier.com/blog/archives/2012/04/password_secu... was in response to last year's breach.
One of the comments[1] on that post has an interesting idea:
> "A server that I use for mail etc., deals with protecting its users from themselves by regularly attempting dictionary and bruteforce attacks upon its own password file. Any passwords that get broken get expired (forcing a change next login) and the user gets a (mostly polite) email explaining why their password was expired and how to avoid it happening again."
Thoughts?
[1] http://www.schneier.com/blog/archives/2012/04/password_secur...
> "A server that I use for mail etc., deals with protecting its users from themselves by regularly attempting dictionary and bruteforce attacks upon its own password file. Any passwords that get broken get expired (forcing a change next login) and the user gets a (mostly polite) email explaining why their password was expired and how to avoid it happening again."
Thoughts?
[1] http://www.schneier.com/blog/archives/2012/04/password_secur...
If they're using pbkdf2, scrypt, bcrypt, etc., there's a much better option than attempting to brute force a database of salted passwords.
You could check a lot (depending on language speed) of common passwords whenever a password is chosen, or during login, when you still have the password in cleartext. When it passes the checks, set a flag for that user so that the check doesn't have to be repeated.
If it takes to long to do during login, have a counter that keeps track of where in a list of passwords the checker is at, and do N more each login.
You could check a lot (depending on language speed) of common passwords whenever a password is chosen, or during login, when you still have the password in cleartext. When it passes the checks, set a flag for that user so that the check doesn't have to be repeated.
If it takes to long to do during login, have a counter that keeps track of where in a list of passwords the checker is at, and do N more each login.
According to [1], the 100 most frequently used passwords are used by 40% of people, and the 500 most frequently used passwords are used by 71% of people. So you can increase security for over half of your users simply by disallowing the 200 most commonly used passwords. This set of passwords is small enough to fit in an in-memory list, so you don't even need to use a database for that.
I don't think there's much need to check for common passwords at each login, provided that none of your users are allowed to use such passwords in the first place. You might get more security by banning IPs that try more than a few of those common passwords in a short time (it's a useful way to detect brute-force attempts), but in fact you should be banning any IP that generates more than a few failed logins in a short time.
[1] http://xato.net/passwords/more-top-worst-passwords/
Another measure I use on my webapps is calculating how many bits of entropy a password contains, and ban anything less than ~40 bits (the threshold depends on the target demographic). An 8-digit number has 10^8 = 26.5 bits of entropy. An 8-char lowercase string has 26^8 = 37.6 bits of entropy (unless it's a common word). A mixed-case 8-char string that also contains numbers has (26+26+10)^8 = 47.6 bits of entropy. This method has the advantage that I don't need to specify annoying rules like "one lowercase, one uppercase, one symbol, etc." Simpler passwords can be allowed if they are long enough, and complex passwords can be banned if they are too short.
I don't think there's much need to check for common passwords at each login, provided that none of your users are allowed to use such passwords in the first place. You might get more security by banning IPs that try more than a few of those common passwords in a short time (it's a useful way to detect brute-force attempts), but in fact you should be banning any IP that generates more than a few failed logins in a short time.
[1] http://xato.net/passwords/more-top-worst-passwords/
Another measure I use on my webapps is calculating how many bits of entropy a password contains, and ban anything less than ~40 bits (the threshold depends on the target demographic). An 8-digit number has 10^8 = 26.5 bits of entropy. An 8-char lowercase string has 26^8 = 37.6 bits of entropy (unless it's a common word). A mixed-case 8-char string that also contains numbers has (26+26+10)^8 = 47.6 bits of entropy. This method has the advantage that I don't need to specify annoying rules like "one lowercase, one uppercase, one symbol, etc." Simpler passwords can be allowed if they are long enough, and complex passwords can be banned if they are too short.
The "unless it's a common word" thing is the problem, really. I tried to do a more Markovian filter at my work and it bombed -- nobody enjoyed trying to get around its entropy restrictions. All it did was assume that your next letter would be predictably "like" your last one, so "password1" would have just one domain change in it. I didn't even go through, e.g., John the Ripper's dictionary to build an optimized order for going through letters.
Ban common words. Ban common words with 1-2 numbers attached to the end. Ban common words with 1-2 letters replaced with similar-looking numbers. But unless your website deals with very sensitive information, I don't think it's an efficient use of resources to try to use a more sophisticated filter. Simply imposing a logarithmic delay between login attempts, and banning IPs / disabling accounts after a certain number of failed attempts, go a long way toward protecting any password that isn't in the top 1000.
"password1" is too easy to brute-force. "pAssword072" is not even in the top 10000, and therefore it is already stronger than at least 98.8% of passwords that are out there.
The idea is to allow weaker passwords for the sake of convenience, but compensate for their weakness by adding protections in other areas. For example, you might require users with weaker passwords to change their passwords more often, disable their accounts more aggressively when a brute-force attack is suspected, or use more rounds of bcrypt to encrypt their passwords. Some of these measures will be helpful even in the case of a database breach.
"password1" is too easy to brute-force. "pAssword072" is not even in the top 10000, and therefore it is already stronger than at least 98.8% of passwords that are out there.
The idea is to allow weaker passwords for the sake of convenience, but compensate for their weakness by adding protections in other areas. For example, you might require users with weaker passwords to change their passwords more often, disable their accounts more aggressively when a brute-force attack is suspected, or use more rounds of bcrypt to encrypt their passwords. Some of these measures will be helpful even in the case of a database breach.
The Visa I was using for billing with Linode had an authorized "test charge" earlier this week, and I had the card number replaced. Now I see this, and it makes me wonder.
My Visa (that I use for Linode) was compromised this week too, someone tried to order ~$100 of stuff from Amazon but my bank rejected the charge and now I have to wait another week before I can make payments again (killed my card...). Probably a coincidence but it makes me wonder too...
Was that credit card number used anywhere else? Given Linode's size, it's not surprising for a few customers to have had coincidental CC compromises recently.
Ugh. Now I am worried about my card too (current linode customer). :/
I wish my CC provider supported 'single use' or 'limit based' generated numbers. I think discover card does?
edit: something like this https://www.discover.com/credit-cards/member-benefits/securi... but for my card would be neat
I wish my CC provider supported 'single use' or 'limit based' generated numbers. I think discover card does?
edit: something like this https://www.discover.com/credit-cards/member-benefits/securi... but for my card would be neat
I'm glad to see Linode is keeping an eye out for its customers, nothing wrong with that.
On one hand, I love Linode. It's a sweet Linux box in the cloud.
On the other hand, I have a hard time using it for anything sensitive. Quotes like "We have implemented all appropriate measures to provide the maximum amount of protection to our customers." somehow don't reassure me.
On the other hand, I have a hard time using it for anything sensitive. Quotes like "We have implemented all appropriate measures to provide the maximum amount of protection to our customers." somehow don't reassure me.
This is how it should be, IMO. Never trust a service entirely. I also use Linode, fwiw.
nice job linode!
so what's a good alternative that offers all of security, reliability, good service, about $40 for 2GB/8-cores, multi-locations, etc?
This incident has got me seriously thinking about switching VPS providers. Does anyone know of a VPS provider that offers two-factor authentication for its management interface?
AWS provides multi-factor authentication, which I've always used quite successfully. https://aws.amazon.com/mfa/
https://gist.github.com/5376298