Fighting Cargo Cult – An Incomplete SSL/TLS Bookmark Collection(daniel.molkentin.net)
daniel.molkentin.net
Fighting Cargo Cult – An Incomplete SSL/TLS Bookmark Collection
https://daniel.molkentin.net/2014/04/21/fighting-cargo-cult-the-incomplete-ssltls-bookmark-collection/
6 comments
With the rise of the cloud, now everybody manages a server. I am curious about how many of the admins actually know what are they doing. If I have to speak for myself, I set up my first ssl by following a guide that seemed to written by reasonably well informed stranger. It was my first exposure to the topic so I could not tell what I am doing until I break things and start reading deeper materials so that I can debug the problems by understanding what's going on. Turns out that the stranger who wrote the tutorial I followed was not well informed at all.
Yes, that's the exact problem that made me write this. What's particularly amazing is the amount of magical cipher suite strings shared throughout the web, most of which do not take in account PFS, or still prioritize RC4. All of that was acceptable at some point in time. Other cipher lists just don't make any sense at all, e.g. first removing a cipher (-RC4), then killing it (!RC4), all in one string with no benefit at all.
Thanks for writing this post, I've bookmarked it for reading later as it has plenty of links and I can see I have a lot more reading to do.
If you edit or update your post, I hope you will Mozilla's excellent "Security/Server Side TLS" page at https://wiki.mozilla.org/Security/Server_Side_TLS. This helped me get up to speed quickly and provided clear examples.
As proof of how good the Mozilla docs are, I tested my personal website using the Qualys test you mention and received an A+ rating!
This beat the A- rating for your site (though I freely admit I'm a total noob in this area - I'm copy/pasting and don't understand much about SSL). I guess this reinforces your point that good documentation is critical, and I hope more people find it at the Mozilla site.
https://www.ssllabs.com/ssltest/analyze.html?d=tombrossman.c...
https://www.ssllabs.com/ssltest/analyze.html?d=daniel.molken...
If you edit or update your post, I hope you will Mozilla's excellent "Security/Server Side TLS" page at https://wiki.mozilla.org/Security/Server_Side_TLS. This helped me get up to speed quickly and provided clear examples.
As proof of how good the Mozilla docs are, I tested my personal website using the Qualys test you mention and received an A+ rating!
This beat the A- rating for your site (though I freely admit I'm a total noob in this area - I'm copy/pasting and don't understand much about SSL). I guess this reinforces your point that good documentation is critical, and I hope more people find it at the Mozilla site.
https://www.ssllabs.com/ssltest/analyze.html?d=tombrossman.c...
https://www.ssllabs.com/ssltest/analyze.html?d=daniel.molken...
That's an excellent reference with good explanations. I'll add it to the list to get away from the strong Ivan bias :-). The reason why I had A- only is that my openssl (Debian) doesn't seem provide all the ciphers required.
It's probably your apache, actually; 2.2 can't do ECDHE, even though the openssl in Debian 7 can.
Sadly, this means IE doesn't get forward secrecy, because it can't do do DHE with RSA keys, only DSS.
Sadly, this means IE doesn't get forward secrecy, because it can't do do DHE with RSA keys, only DSS.
Yet Another Test Script: https://github.com/jvehent/cipherscan
It uses openssl to find out the cipher priorities, PFS keys and a few others things from a target site.
I wrote it because, while I love SSLLabs and Ivan's work in general, I find it too slow to scan a setup I'm working on many times. And I like doing stuff in the terminal.
A very nice python script to check your server configuration: https://github.com/iSECPartners/sslyze
Thanks, added!
A lot of sys admins are there by default. I would consider myself one of these - I would love to be able off load this work onto someone else who knows more than me, but there just isn't anyone.
Essentially this comes down to pressuring distros and server vendors do their homework finally ship with good examples/defaults. E.g. Microsoft IIS (!) has OCSP stapling enabled by default since ages. Apache? Most people still run 2.2, which isn't capable of OCSP stapling at all. Nginx is in a similar position.
That said, the more sysadmins rely on <s>rotten</s>well-proven "Enterprise Linuxes" and "LTS" versions with old libraries and servers, the more security expertise is required from sysadmins to decide where to deviate from the distros default packages to meet current best practices.
On the other hand, security is a moving target and knowing your (Open)SSL setup is as important as e.g. knowing your RoR setup. It's an inconvenient truth, because it requires learning new stuff. I don't see any alternative though, that's why I compiled this material.
Finally, a remark on the "offloading" part: Security is the single thing where delegation becomes hard because it means delegating trust, as in: your private SSL/TLS keys. And that's quite some trust to delegate.
That said, the more sysadmins rely on <s>rotten</s>well-proven "Enterprise Linuxes" and "LTS" versions with old libraries and servers, the more security expertise is required from sysadmins to decide where to deviate from the distros default packages to meet current best practices.
On the other hand, security is a moving target and knowing your (Open)SSL setup is as important as e.g. knowing your RoR setup. It's an inconvenient truth, because it requires learning new stuff. I don't see any alternative though, that's why I compiled this material.
Finally, a remark on the "offloading" part: Security is the single thing where delegation becomes hard because it means delegating trust, as in: your private SSL/TLS keys. And that's quite some trust to delegate.
> knowing your (Open)SSL setup is as important as e.g. knowing your RoR setup
Very true, and yet it's so much harder to know your OpenSSL or other security-related setup.
You learn your Rails setup well enough to make your application work, and hopefully well enough to make it performant. If you miss either of those goals, it's obvious to you. You know something's broken, and you grind away until you fix it.
Not so with security. Your system can be "working" by all outward appearances yet be riddled with vulnerabilities. And you won't know it, so you won't see any reason to go and learn more. Nor would you know what you don't know, or where you need to learn more. That's the scary part.
Very true, and yet it's so much harder to know your OpenSSL or other security-related setup.
You learn your Rails setup well enough to make your application work, and hopefully well enough to make it performant. If you miss either of those goals, it's obvious to you. You know something's broken, and you grind away until you fix it.
Not so with security. Your system can be "working" by all outward appearances yet be riddled with vulnerabilities. And you won't know it, so you won't see any reason to go and learn more. Nor would you know what you don't know, or where you need to learn more. That's the scary part.
There is so much I agree with in your post. The more I learn in this area the less sure I am that I am doing the right things.
Okay, I understand what he means about cargo culting, and that to really make your server secure you have to actually know what you're doing.
But okay, let's be realistic here. Let s look at the real thing that just happened to me.
I went and used the awesome tester at ssllabs.com that OP recommends.
It told me that my server doesn't support newer cipher protocols that it should support, and that it doesn't support forward secrecy.
So, okay, I start googling on how to fix this. It looks like supporting forward secrecy is also an issue of cipher protocols, if I support the proper cipher protocols I'll support forward secrecy.
Okay, how am I going to figure out which cipher protocols to configure as supported in apache? I'm to google and copy and paste someone elses apache configuration. That is, I'm going to cargo cult it.
What else is possibly realistic for me to do? How many hours would it take for me to actually understand myself which ciphers are secure, which are preferred, and why? Not just how many hours would it take now, but how many _continual_ hours to keep up with developments?
We all can't be experts at this level in this area -- that level of expertise in security is pretty much a role all by itself. And we can't all even have such experts on staff. We've _got_ to rely on external experts, anything else is unrealistic.
Call that 'cargo culting' all you want, but what I need is an expert to tell me what my apache config should be -- sure I'm going to read up to get enough background to basically understand what they're talking about, but there's no way I'm going to become enough of an expert to make this determination on my own.
No?
Many use apache httpd. Seems to me apache httpd should ship with proper SSL configuration, and apache httpd team should provide notification channels for people to change configuration on their already-installed apache when best practices change. Figuring this out can't be a task for each individual 'customer', for basic use cases and routine needs, it should be the task of the 'vendor', no?
But okay, let's be realistic here. Let s look at the real thing that just happened to me.
I went and used the awesome tester at ssllabs.com that OP recommends.
It told me that my server doesn't support newer cipher protocols that it should support, and that it doesn't support forward secrecy.
So, okay, I start googling on how to fix this. It looks like supporting forward secrecy is also an issue of cipher protocols, if I support the proper cipher protocols I'll support forward secrecy.
Okay, how am I going to figure out which cipher protocols to configure as supported in apache? I'm to google and copy and paste someone elses apache configuration. That is, I'm going to cargo cult it.
What else is possibly realistic for me to do? How many hours would it take for me to actually understand myself which ciphers are secure, which are preferred, and why? Not just how many hours would it take now, but how many _continual_ hours to keep up with developments?
We all can't be experts at this level in this area -- that level of expertise in security is pretty much a role all by itself. And we can't all even have such experts on staff. We've _got_ to rely on external experts, anything else is unrealistic.
Call that 'cargo culting' all you want, but what I need is an expert to tell me what my apache config should be -- sure I'm going to read up to get enough background to basically understand what they're talking about, but there's no way I'm going to become enough of an expert to make this determination on my own.
No?
Many use apache httpd. Seems to me apache httpd should ship with proper SSL configuration, and apache httpd team should provide notification channels for people to change configuration on their already-installed apache when best practices change. Figuring this out can't be a task for each individual 'customer', for basic use cases and routine needs, it should be the task of the 'vendor', no?
This is yet another manifestation of the dark side of open source movement: the expectation that the generosity of the few must always support the comfort of the many. More than once I have commented that the only missing feature of GPL is that all free software must be distributed in source code, and compiled/build from scratch in every system it is to be deployed. Binary distribution requires a parallel form of licensing with appropriate fees/royalties.
Your case would be easily be solved if you were willing to pay an expert provider for their know how in implementing this types of solution. This solves reasonably well the problem of reliability (customer is unable to tell a lemon from the real thing, but is at least able to demand a refund or sue in case of breach of contract from the part of the provider).
This does not have to be expensive either. For a couple USD$100's you could have access to standard configurations, designed to meet the common needs of most customers. Or you can pay premium (a.k.a. consulting) fees to have a tailor made solution made for your particular needs.
Your case would be easily be solved if you were willing to pay an expert provider for their know how in implementing this types of solution. This solves reasonably well the problem of reliability (customer is unable to tell a lemon from the real thing, but is at least able to demand a refund or sue in case of breach of contract from the part of the provider).
This does not have to be expensive either. For a couple USD$100's you could have access to standard configurations, designed to meet the common needs of most customers. Or you can pay premium (a.k.a. consulting) fees to have a tailor made solution made for your particular needs.
Well, I don't need to pay $200, I can also Google around for standard configurations designed to meet the common needs of most customers -- plenty of people are more than happy to share all sorts of info on proper SSL configuration for free. As for instance, the OP.
But if I just find a configuration online and use it, am I 'cargo culting' as the OP warns me against? But then, am I doing the same thing if I pay someone for it instead of finding it online for free, am I still just 'cargo culting'? $200 or not, I'm still just taking what someone else gives me and plugging it in. I guess in either case it depends on who I get it from, and how I am deciding their reliability.
I don't think it's the 'dark side of open source' to suggest that apache would be doing a service to it's users by providing out-of-the-box config that's actually secure. I realize they don't 'owe' it to me, as they 'owe' me nothing at all, including continuing to provide httpd at all. But let's say httpd was completely unstable out of the box, crashing all the time, unless you used expert knowledge to configure it to be stable, possibly by paying an expert $200. Would it be the 'dark side of open source' to suggest that made it less high-quality software than it could be, that to be quality software it needs to be stable for most common use cases out of the box?
In 2014, I think expecting software that supports SSL to be actually secure out of the box for, as you say, 'common needs of most customers', is just the same as expecting it to be reasonably stable. Doesn't mean it will be, doesn't mean the open source developers 'owe' us anything, but users making their expectations and priorities clear is part of what influences developer priorities too.
But you know what, yeah, I'm going to go there, if in 2014 you are supplying software to users (whether open source or not, whether free-as-in-beer or not) that supports TLS/SSL, or any other crypo, then you _do_ have some responsibility to supply actually secure software, or you are doing a disservice to your users (whether or not they are paying you) and to the internet at large. I don't think this is actually a very controversial statement. Nobody has to supply crypto software, open source or proprietary, for free or for $. But if you are, then, yeah, you have some responsibilities to do it right. We can certainly disagree on what 'do it right' means, but few would disagree with the basic sentiment.
But if I just find a configuration online and use it, am I 'cargo culting' as the OP warns me against? But then, am I doing the same thing if I pay someone for it instead of finding it online for free, am I still just 'cargo culting'? $200 or not, I'm still just taking what someone else gives me and plugging it in. I guess in either case it depends on who I get it from, and how I am deciding their reliability.
I don't think it's the 'dark side of open source' to suggest that apache would be doing a service to it's users by providing out-of-the-box config that's actually secure. I realize they don't 'owe' it to me, as they 'owe' me nothing at all, including continuing to provide httpd at all. But let's say httpd was completely unstable out of the box, crashing all the time, unless you used expert knowledge to configure it to be stable, possibly by paying an expert $200. Would it be the 'dark side of open source' to suggest that made it less high-quality software than it could be, that to be quality software it needs to be stable for most common use cases out of the box?
In 2014, I think expecting software that supports SSL to be actually secure out of the box for, as you say, 'common needs of most customers', is just the same as expecting it to be reasonably stable. Doesn't mean it will be, doesn't mean the open source developers 'owe' us anything, but users making their expectations and priorities clear is part of what influences developer priorities too.
But you know what, yeah, I'm going to go there, if in 2014 you are supplying software to users (whether open source or not, whether free-as-in-beer or not) that supports TLS/SSL, or any other crypo, then you _do_ have some responsibility to supply actually secure software, or you are doing a disservice to your users (whether or not they are paying you) and to the internet at large. I don't think this is actually a very controversial statement. Nobody has to supply crypto software, open source or proprietary, for free or for $. But if you are, then, yeah, you have some responsibilities to do it right. We can certainly disagree on what 'do it right' means, but few would disagree with the basic sentiment.
I've essentially answered this in https://news.ycombinator.com/item?id=7620756.
Basically, this cannot be the sole task of Apache, as they rely on OpenSSL via mod_ssl. Some parts are apache specific (OCSP stapling), but others, just like cipher suites, purely depend on OpenSSL. But I agree that Distributors (as the "Vendors") need to come up with a suitable solution.
Basically, this cannot be the sole task of Apache, as they rely on OpenSSL via mod_ssl. Some parts are apache specific (OCSP stapling), but others, just like cipher suites, purely depend on OpenSSL. But I agree that Distributors (as the "Vendors") need to come up with a suitable solution.
[deleted]
I don't think that's true. Copy-and-pasting an apache config because it does what you want is different from copy-and-pasting an apache config and hoping that it does what you want.
I don't think this article is really helpful at all. The author attacks blog posts that state the current best-practices, but then goes on to recommend several books that are even more out of date than the blog posts. (Some of the books are 'theory' books and so presumably remain 'current' for longer, but not all of them).
Second, instead of allowing sysadmins to find and follow simple, well-researched best-practices, the author instead wants each person to thoroughly research the annals of cryptography in order to then come to their 'own' conclusion. Most sysadmins won't do this. Or they will miss something or make rookie mistakes (this why you're always told not to roll your own cryptography). To put it another way: "you should not roll your own choices about which cryptography to use".
So for the other 99% who just need to get it right and move on: find the current best practices from a reputable source (like Qualys) and use those.
Second, instead of allowing sysadmins to find and follow simple, well-researched best-practices, the author instead wants each person to thoroughly research the annals of cryptography in order to then come to their 'own' conclusion. Most sysadmins won't do this. Or they will miss something or make rookie mistakes (this why you're always told not to roll your own cryptography). To put it another way: "you should not roll your own choices about which cryptography to use".
So for the other 99% who just need to get it right and move on: find the current best practices from a reputable source (like Qualys) and use those.
>The author attacks blog posts that state the current best-practices
No, I'm attacking the fact that people blingly follow blog posts that have been, at some point, what their author believed were best practices.
> But then goes on to recommend several books that are even more out of date than the blog posts.</cite>
Which is fine, as long as they are read for what they are supposed to be: Either introductions or specializing on a specific topic. I wouldn't have chosen them otherwise.
> Second, instead of allowing sysadmins to find and follow simple, well-researched best-practices, the author instead wants each person to thoroughly research the annals of cryptography in order to then come to their 'own' conclusion
It is up to your own self-conception as a sysadmin as to how deep you want to dive. When interviewing (non-junior grade) sysadmins, I challenge them on security knowledge just as much as other skills. And I know I'm not alone with that. A certain degree of security awareness is not a "nice to have" typoe additional skill. It's vital for everyone that has machines connected to the internet. How far he/she can dive is solely limited by the economics and time constrains. Which is why sensible defaults are needed from vendors and distros (see my other response).
> Most sysadmins won't do this.
Which is a real problem, and again, there should be some effort remedying this, but currently there isn't. That's why I plea to sit down and at least learn about the basics. What "the basics" are obviously depends greatly on your educational background, but if AES, RC4 and PFS do not ring a bell, and you are a professional sysadmin who runs SSL-secured web servers, you are not worth your money.
> (this why you're always told not to roll your own cryptography)
Nobody said anything about rolling your own cryptography. But if you at least have read one of Ivan's books, you can at least make a /qualified decision/ on how credible a config proposed in a "random blog post" is, without having to come up with the complete solution by yourself.
> find the current best practices from a reputable source (like Qualys) and use those.
Which is ok -- if you run a small setup and you at least use SSLLabs to verify your setup. The more responsibility you have, and the more security is required, the more you should dive in and have a qualified opinion on how your SSL/TLS setup should look like.
No, I'm attacking the fact that people blingly follow blog posts that have been, at some point, what their author believed were best practices.
> But then goes on to recommend several books that are even more out of date than the blog posts.</cite>
Which is fine, as long as they are read for what they are supposed to be: Either introductions or specializing on a specific topic. I wouldn't have chosen them otherwise.
> Second, instead of allowing sysadmins to find and follow simple, well-researched best-practices, the author instead wants each person to thoroughly research the annals of cryptography in order to then come to their 'own' conclusion
It is up to your own self-conception as a sysadmin as to how deep you want to dive. When interviewing (non-junior grade) sysadmins, I challenge them on security knowledge just as much as other skills. And I know I'm not alone with that. A certain degree of security awareness is not a "nice to have" typoe additional skill. It's vital for everyone that has machines connected to the internet. How far he/she can dive is solely limited by the economics and time constrains. Which is why sensible defaults are needed from vendors and distros (see my other response).
> Most sysadmins won't do this.
Which is a real problem, and again, there should be some effort remedying this, but currently there isn't. That's why I plea to sit down and at least learn about the basics. What "the basics" are obviously depends greatly on your educational background, but if AES, RC4 and PFS do not ring a bell, and you are a professional sysadmin who runs SSL-secured web servers, you are not worth your money.
> (this why you're always told not to roll your own cryptography)
Nobody said anything about rolling your own cryptography. But if you at least have read one of Ivan's books, you can at least make a /qualified decision/ on how credible a config proposed in a "random blog post" is, without having to come up with the complete solution by yourself.
> find the current best practices from a reputable source (like Qualys) and use those.
Which is ok -- if you run a small setup and you at least use SSLLabs to verify your setup. The more responsibility you have, and the more security is required, the more you should dive in and have a qualified opinion on how your SSL/TLS setup should look like.
Just to be clear, my books mentioned in the blog post (Bulletproof SSL and TLS and OpenSSL Cookbook) are most certainly not out of date. In fact, they are a rare example of books that are continuously maintained. I pledged to maintain them for as long as there are people interested in reading them!
Well, "don't roll your own crypto" advice has recently been compared to abstinence-only birth control. It sounds well in theory, but in practices is prone to not work at all.
This is because most defects in security come from application developers assembling poorly understood third party solutions in ways that the original authors never intended or even imagined to be possible. Best practices and secure defaults can only take you so far, so there is a genuine need for minimum competence standards, even if those are informal and not regulated.
This is because most defects in security come from application developers assembling poorly understood third party solutions in ways that the original authors never intended or even imagined to be possible. Best practices and secure defaults can only take you so far, so there is a genuine need for minimum competence standards, even if those are informal and not regulated.
Following incorrect guidelines is "cargo cult" now? Or are the guidelines themselves cargo cult? I'm confused.
All "cargo cult" means is that you're doing something associated with an outcome that you want, rather than understanding the mechanism by which what you're doing would lead to the desired outcome.
A common example: towns that have a lot of tourists have a lot of parking. Therefore if we tear down buildings to make more parking lots, we'll increase tourism!
I'm pretty sure that what 99.9% of developers (including me) are doing with SSL is cargo cult.
A common example: towns that have a lot of tourists have a lot of parking. Therefore if we tear down buildings to make more parking lots, we'll increase tourism!
I'm pretty sure that what 99.9% of developers (including me) are doing with SSL is cargo cult.
Following an incorrect tutorial that is supposed to bestow security is as useful as following ritualistic acts intended to bestow wealth.
There are a lot of steps and motions included in many resources, without a good reason behind them, that one follows. You basically have to assume a degree of faith that they have a purpose and actually achieve what the author purports they achieve.
There are a lot of steps and motions included in many resources, without a good reason behind them, that one follows. You basically have to assume a degree of faith that they have a purpose and actually achieve what the author purports they achieve.
TLS 1.0 was created in 1999, and has approximately 97.7% website support worldwide. The only modern browser that doesn't support TLS 1.0 is IE6. Please, do the public a favor and remove the name "SSL" from your list.
Also, I get that you probably just found out what a cargo cult is, but there's really no purpose in telling people about the phrase; it doesn't help them stay more secure, and it's basically just pop psychology applied to technology. "Current guides for securing your TLS servers" would make a more accurate title.
Also, I get that you probably just found out what a cargo cult is, but there's really no purpose in telling people about the phrase; it doesn't help them stay more secure, and it's basically just pop psychology applied to technology. "Current guides for securing your TLS servers" would make a more accurate title.
About the Cargo Cult thing: Fair enough. I'm not a native speaker myself, but thought that it may be a well-enough-known idiom.
Anyway, SSL is still what people know it under (plus, according to Wikipedia, TLS support was only added in Java 7, and there are still many Java 6 setups around which have to put up with SSL 3). So I'll stick with SSL/TLS for the time being.
Anyway, SSL is still what people know it under (plus, according to Wikipedia, TLS support was only added in Java 7, and there are still many Java 6 setups around which have to put up with SSL 3). So I'll stick with SSL/TLS for the time being.
Re: "Cargo Cult" as a well-enough-known idiom:
It's basically a shibboleth for "Are you a fan of Richard Feynman". I doubt very many people have actually heard of it directly, but his "Cargo Cult Science" commencement address (http://neurotheory.columbia.edu/~ken/cargo_cult.html) is moderately famous.
It's basically a shibboleth for "Are you a fan of Richard Feynman". I doubt very many people have actually heard of it directly, but his "Cargo Cult Science" commencement address (http://neurotheory.columbia.edu/~ken/cargo_cult.html) is moderately famous.
IE6 supports TLS 1.0
TLS 1.0 is not enabled by default in IE6, so for practical purposes, IE6 does not support TLS 1.0.
I have no idea what I am doing. Is my Digital Ocean - Default Ubuntu based server safe? I know I login via the shell using ssh from my OS-X and DO does not administrate the droplet I guess. Any advice on where I can educate myself?
We have a couple of resources to help you with that in our community:
https://www.digitalocean.com/community/questions/how-do-you-...
https://digitalocean.com/community/tags/security
Thanks! Moisey Cofounder DigitalOcean
https://www.digitalocean.com/community/questions/how-do-you-...
https://digitalocean.com/community/tags/security
Thanks! Moisey Cofounder DigitalOcean