Bit.ly compromised – More Details(blog.bitly.com)
blog.bitly.com
Bit.ly compromised – More Details
http://blog.bitly.com/post/85260908544/more-detail
48 comments
You'd be surprised how much "PHP login" tutorial use md5 for password encryption. And then people are surprised these devs dont known better? it's because of all the mis-eductive content the web cant get rid of.
Also, many startups consider security to be a waste of time, since you don't need a secure site to generate revenue (arguably, appearing secure has more value in that sense than actually being secure.)
Also I still see people now and then arguing that salted md5 is perfectly secure and that using a slower hash or any hash at all either doesn't add value or actually harms value. So they may not know better, or they may know better and simply not care.
Also I still see people now and then arguing that salted md5 is perfectly secure and that using a slower hash or any hash at all either doesn't add value or actually harms value. So they may not know better, or they may know better and simply not care.
> Also, many startups consider security to be a waste of time
Do you have evidence to support this?
I wouldn't say that security is considered a waste of time.
It is, however is a trade-off[1] between money, time, and other resources, things that, generally speaking, start-ups tend to be short on.
I worked for a startup that provided multifactor authentication using biometrics (amongst others), and almost all of our business came after publicized (and not publicized) breaches, from both large and small firms.
[1]https://www.schneier.com/essay-155.html
Do you have evidence to support this?
I wouldn't say that security is considered a waste of time.
It is, however is a trade-off[1] between money, time, and other resources, things that, generally speaking, start-ups tend to be short on.
I worked for a startup that provided multifactor authentication using biometrics (amongst others), and almost all of our business came after publicized (and not publicized) breaches, from both large and small firms.
[1]https://www.schneier.com/essay-155.html
We may be talking about the same phenomenon from different points of view. I have evidence but I really don't want to name names - but anecdotally, I have pointed out massive issues (like not even escaping user-provided variables in inline SQL queries) to clients, and seen it dismissed as less important than just getting live.
Maybe my comment was a bit too biased and hyperbolic, but I've certainly encountered that attitude.
Maybe my comment was a bit too biased and hyperbolic, but I've certainly encountered that attitude.
I once registered wrongtutorial.com to call out incorrect and dangerous tutorials. I couldn't write enough content to get it off the ground. Do you think it'd still be a good idea?
A more general form of tutorial review/correction site would be more interesting.
Actually, a browser add-on to load commentary/annotations for known tutorials on the fly might be nice. All sorts of practical problems that would have to be overcome, though (like changing content and getting people to use the add-on in the first place).
Actually, a browser add-on to load commentary/annotations for known tutorials on the fly might be nice. All sorts of practical problems that would have to be overcome, though (like changing content and getting people to use the add-on in the first place).
> I couldn't write enough content to get it off the ground.
Poke around w3schools. That should give you more than enough initial grist for your mill.
Poke around w3schools. That should give you more than enough initial grist for your mill.
Definetly! I love this idea!
It's still a bit scary that they're apparently using some ad-hoc KDF (key derivation function) built from bcrypt and HMAC. bcrypt is a decent KDF, but there are several ways to make a bad KDF from a combination of bcrypt and HMAC. For instance, concatenating bcrypt(salt, pw) and HMAC-SHA512(salt, pw) makes guessing passwords way too fast.
I'm hoping that all they've done is use bcrypt(salt, HMAC-SHA256(salt, pw)) to get around the bcrypt 72 character limit. However, using scrypt would have been better.
I'm hoping that all they've done is use bcrypt(salt, HMAC-SHA256(salt, pw)) to get around the bcrypt 72 character limit. However, using scrypt would have been better.
why does concatenating bcrypt(salt, pw) with HMAC(salt, pw) make guessing fast?
Intuitively it seems like you'd still need to guess the bcrypt portion, making it no worse than just bcrypt.
Intuitively it seems like you'd still need to guess the bcrypt portion, making it no worse than just bcrypt.
Why not just ignore the bcrypt portion? "Intuitively" it seems to me like you could just un-concatonate the two strings and guess at them separately, working on whichever you know to be easier.
Hopefully, the 'salt' passed to HMAC-SHA256 is a pepper (preferably stored outside the database), not the same user salt as what bcrypt uses. As for scrypt, it definitely has great potential, but I still feel it's too new and not as well understood to be used 100% correctly in production by most folks (see also http://blog.ircmaxell.com/2014/03/why-i-dont-recommend-scryp...).
What's wrong with bcrypt?
Salted MD5/SHA is basically the standard even though it's awful, and the flaws in MD5's algorithm don't really apply to the password case.
The flaws in MD5 don't really apply to passwords, but the flaws in salted MD5/SHAx sure do.
It's not scary, it's the truth, and it's bad. But, what can you do?
I've never been clear about what value-add registration brings to a link-shortener, so I've just used bit.ly without registration.
I'm kind of happy about that now.
I'm kind of happy about that now.
For the individual user, there probably isn't.
For companies, marketers, etc, bitly, and other url shortners provide data and analytics ontop of the links, allowing more insight into link shares & clicks.
When you're interested in tracking the data from a link, or series of links, having a persistent account is necessary.
For companies, marketers, etc, bitly, and other url shortners provide data and analytics ontop of the links, allowing more insight into link shares & clicks.
When you're interested in tracking the data from a link, or series of links, having a persistent account is necessary.
it allows you to track stats more easily. for example, you can link bit.ly to Buffer and get a unique bit mark for each link/platform pair. then you can track engagement separately for each platform.
Couldn't you just have an IFTTT pull anything you bitly into a bucket of links to watch stats on? Why does bitly need to know who you are, or what links you submitted, if you can just remember (maybe through automation) which links are yours, and bitly or buffer can monitor each link individually for stats?
My guess is that logins probably help bitly track usage patterns, improve the product, and keep an eye out for ways to increase their value by measuring engagement with their userbase. Maybe it was just "everyone's adding a social layer" peer pressure. Not sure.
There are security costs to "use passwords everywhere, for everything," but they are more diffuse. We've ended up with a system where security best practices would recommend using hundreds of distinct passwords that are each too complex to memorize (so you can dump them in a locker or support one login services, each creating a single point of failure, or you can use a similar password or mental password algorithm for each site, it's all tradeoffs).
I don't think there's an obviously better solution, I think it's just a tragedy of the commons we were probably inevitably going to trip over sooner or later.
My guess is that logins probably help bitly track usage patterns, improve the product, and keep an eye out for ways to increase their value by measuring engagement with their userbase. Maybe it was just "everyone's adding a social layer" peer pressure. Not sure.
There are security costs to "use passwords everywhere, for everything," but they are more diffuse. We've ended up with a system where security best practices would recommend using hundreds of distinct passwords that are each too complex to memorize (so you can dump them in a locker or support one login services, each creating a single point of failure, or you can use a similar password or mental password algorithm for each site, it's all tradeoffs).
I don't think there's an obviously better solution, I think it's just a tragedy of the commons we were probably inevitably going to trip over sooner or later.
What percentage of the internet would even understand what you just suggested, much less know how to do it?
Also, you'd have to see a massive amount of value in not signing up to Bitly to offset the hours you'd spend creating a vastly inferior hack job on your own.
Also, you'd have to see a massive amount of value in not signing up to Bitly to offset the hours you'd spend creating a vastly inferior hack job on your own.
> "[Your ideas are] vastly inferior hack jobs, 0/10 stars..." [ very roughly :) ]
All I'm claiming is that bitly could have offered stats without a login system. IFTTT was just a proof of that concept, a gratuitous example, not some idealized form of stat tracking that everyone should run out and implement.
You're probably right that the solution I thought of in around ten seconds sucks. I've spent more time on worse ideas. My real point is that login systems, despite their ubiquity, have their own share of downsides. Some of these downsides should be painfully evident today given the article.
All I'm claiming is that bitly could have offered stats without a login system. IFTTT was just a proof of that concept, a gratuitous example, not some idealized form of stat tracking that everyone should run out and implement.
You're probably right that the solution I thought of in around ten seconds sucks. I've spent more time on worse ideas. My real point is that login systems, despite their ubiquity, have their own share of downsides. Some of these downsides should be painfully evident today given the article.
Hopefully this will cause companies to think twice about implementing someone else's link shortening service within their software solution. I've done several reviews of software solutions which have used bit.ly and other shortening services for things like providing a link to corporate identity portals.
It's quite common for a company to want to use their own identity portal for employee authentication with a particular piece of software and a few companies implement this via giving users shortened links which resolve to the identity portal location. They will hardcode the request to bit.ly & then give the users the token to enter when the software starts. It will then send the full bit.ly request and get a destination location back for the portal.
In every case I've seen this, I've written it as a high risk finding since neither the software vendor nor the company using the solution has ever done an audit of the security controls at bit.ly or the other link shortening services. If a link shortening company like bit.ly is compromised & a bad guy is able to modify link destinations then it's pretty much over in terms of security for a software solution implementing this model. There also tends to be some issues in terms of implementing public key or certificate pinning when using a third-party but I'll save that for another day.
It's quite common for a company to want to use their own identity portal for employee authentication with a particular piece of software and a few companies implement this via giving users shortened links which resolve to the identity portal location. They will hardcode the request to bit.ly & then give the users the token to enter when the software starts. It will then send the full bit.ly request and get a destination location back for the portal.
In every case I've seen this, I've written it as a high risk finding since neither the software vendor nor the company using the solution has ever done an audit of the security controls at bit.ly or the other link shortening services. If a link shortening company like bit.ly is compromised & a bad guy is able to modify link destinations then it's pretty much over in terms of security for a software solution implementing this model. There also tends to be some issues in terms of implementing public key or certificate pinning when using a third-party but I'll save that for another day.
Agree . as a individual user's point of view, me too don't understand signup thing for a url shortner .
Link-shorteners are a cancer. We have enough of a problem with historical links going dead, without interposing a third party.
When bit.ly goes down, what happens to all their links?
When bit.ly goes down, what happens to all their links?
True, but for example if I'm posting an article I wrote it's much more efficient to save space using my erclgrgn.us short link because it's usually ~20 chars which is less than half of what the full URL is.
Unfortunately, sometimes brevity is a necessity.
edit: original comment mentioned twitter, then I remembered twitter automatically reduces a URL's chars to 20 or so.
Unfortunately, sometimes brevity is a necessity.
edit: original comment mentioned twitter, then I remembered twitter automatically reduces a URL's chars to 20 or so.
Yeah, or websites could just allow actual links. I don't see why hackernews (and twitter, etc.) can't just render <a href="http://somereallylongdomain.com/stupidlylongpath/?what=thisi... short!</a>
Because of griefer trolls. Forcing links to show the destination is basically 0 effort on the system side and pretty much completely undermines them.
(that only answers the HN part, I'm guessing that is much of the reasoning, I don't know it for a fact)
(that only answers the HN part, I'm guessing that is much of the reasoning, I don't know it for a fact)
By that rationale, any website is a cancer.
Chef supports encrypted data-buckets where you decrypt only on the server. Puppet supports hiera data with encryption as well (via plugins). There's no reason anyone should be storing password information in source, especially API keys which I'm guessing this was (S3 information more than likely?). I think about the extent of private information I'd be okay with storing in source is DB user/pass information because that will generally require compromising the machine more generally first and by that point they already had access to your DB even if the password/username wasn't stored in source.
Essentially this will serve as a wake-up call to the Bitly guys and hopefully they'll take some steps to deploy this stuff via CM or similar in the future.
Essentially this will serve as a wake-up call to the Bitly guys and hopefully they'll take some steps to deploy this stuff via CM or similar in the future.
I read it a bit differently. I don't think they store their credentials directly in their source code repository. I think they are saying they have a separate repository specifically for credentials.
It seems like a bit of a strange way to manage sensitive information, but not really any worse then a lot of other things I've seen.
It seems like a bit of a strange way to manage sensitive information, but not really any worse then a lot of other things I've seen.
For security stuff, if you can't find a well-tested implementation to re-use, I always suggest referring to the OWASP Cheat Series [1] - there's a nice one just for password storage [2].
[1] https://www.owasp.org/index.php/OWASP_Cheat_Sheet_Series [2] https://www.owasp.org/index.php/Password_Storage_Cheat_Sheet
[1] https://www.owasp.org/index.php/OWASP_Cheat_Sheet_Series [2] https://www.owasp.org/index.php/Password_Storage_Cheat_Sheet
Is the root issue here that they stored remote server database passwords in their source code?
I'm always hesitant to have source code with hard coded passwords, I know at some point you have to have them "somewhere" but anything you can do to reduce attack vectors for them is helpful.
What do people think should be best practices for storing sensitive configuration data? I know some people will have just a config repo with very limited access but that still seems dangerous at scale when that ends up being a lot of people etc.
I'm always hesitant to have source code with hard coded passwords, I know at some point you have to have them "somewhere" but anything you can do to reduce attack vectors for them is helpful.
What do people think should be best practices for storing sensitive configuration data? I know some people will have just a config repo with very limited access but that still seems dangerous at scale when that ends up being a lot of people etc.
I have yet to find any solution to this problem that I like. A separate config repo is least bad of the various bad options. The others have all been error-prone and failed to play nicely with version control.
If someone could make git store all on-disk data in encrypted form with an ssh-agent-like solution for on-the-fly (de|en)cryption, that would seem ideal.
If someone could make git store all on-disk data in encrypted form with an ssh-agent-like solution for on-the-fly (de|en)cryption, that would seem ideal.
> If someone could make git store all on-disk data in encrypted form with an ssh-agent-like solution for on-the-fly (de|en)cryption, that would seem ideal.
You could probably roll your own. A decent middleground would be to keep the data in the central repo encrypted, but keep it decrypted on the engineers' workstations. Though a better approach is just to not check in sensitive values at all.
You could probably roll your own. A decent middleground would be to keep the data in the central repo encrypted, but keep it decrypted on the engineers' workstations. Though a better approach is just to not check in sensitive values at all.
"Roll your own" is exactly what causes trouble. It's never right and always painful. We need a general, widely-vetted solution that doesn't make me want to yank my hair out.
"Don't check in sensitive values" is not an option. Credentials must be kept track of and maintained somewhere. Storing them in an ad-hoc manner without version control doesn't just invite trouble, it demands it.
We have password managers for desktops and mobile devices. We need an equivalent for servers that accounts for the fact that the credentials must be highly available, multiple people must be able to use them (whether or not they get to directly view them), some of them will be technicians with limited training following someone else's procedures, and regardless of knowledge and skill, there will be half-asleep people taking emergency actions at 3AM when shit has hit the fan.
"Don't check in sensitive values" is not an option. Credentials must be kept track of and maintained somewhere. Storing them in an ad-hoc manner without version control doesn't just invite trouble, it demands it.
We have password managers for desktops and mobile devices. We need an equivalent for servers that accounts for the fact that the credentials must be highly available, multiple people must be able to use them (whether or not they get to directly view them), some of them will be technicians with limited training following someone else's procedures, and regardless of knowledge and skill, there will be half-asleep people taking emergency actions at 3AM when shit has hit the fan.
I wonder if AWS allows security-group-controlled access to databases, similar to the way you can only allow communication between two ec2 instances based on their security groups.
We use git-crypt (https://github.com/AGWA/git-crypt) in our ansible repo for credentials. Credentials are stored as variable YAML files so anyone can work with them in templates, but only our deploy box and SRE team can encrypt/decrypt the credential files themselves. Generally works pretty well, but you do need to remember to run credential deploys on new boxes. Eventually we'll probably move to a HSM (hardware security module) for the actual storage, but those are pricey (especially since you need two of them).
Some people prefer to keep the keys on specialized servers when a) the need for security is especially acute, and b) the project has the budget.
For more day-to-day stuff, environment variables on the server are an option. If you want to be able to provision a new server 100% automatically and keep the credentials out of source control, you have a few options. You can add a quick interactive phase to the deploy script, wherein the engineer must enter the key manually. Or the engineer could enter a value which is then passed to a KDF.
For more day-to-day stuff, environment variables on the server are an option. If you want to be able to provision a new server 100% automatically and keep the credentials out of source control, you have a few options. You can add a quick interactive phase to the deploy script, wherein the engineer must enter the key manually. Or the engineer could enter a value which is then passed to a KDF.
What about storing the credentials in environment?
"Security Team"?.
I'm honestly curious. I've never seen startups that aren't security startups hiring a security team. Is this a real team or just their developers doing an extra job?
I'm honestly curious. I've never seen startups that aren't security startups hiring a security team. Is this a real team or just their developers doing an extra job?
I don't fault them for it if they don't. "The Bitly security team learned of the potential compromise..." sounds much better than "our programmer, Joe, got an email..."
Good guy Joe. Developer by day, security team by night.
Preferably even the smallest company would have at least a designated "security guy", even if they don't spend most of their time directly on security issues. If there are two, there's your "security team".
Reality, of course, is that most companies don't give much thought to security at all until it bites them in the ass, and security competency is rare enough that even a designated "security guy" could be worse than useless.
Reality, of course, is that most companies don't give much thought to security at all until it bites them in the ass, and security competency is rare enough that even a designated "security guy" could be worse than useless.
Missing step: Asymmetric encryption of off-site backups with passphrase-protected private keys kept offline.
Are they saying the passwords were hashed with MD5 before January? Because that'd be a bit scary.