Clojure web security is worse than you think(hackworth.be)
hackworth.be
Clojure web security is worse than you think
https://hackworth.be/2014/03/26/clojure-web-security-is-worse-than-you-think/
14 comments
I watched that talk yesterday and it was great, his list of issues and concerns are very justified but I wasn't excited about the proposed solutions: "just pick one templating language", "everyone use a (heavy, highly opinionated) framework", etc.
They struck me as both unrealistic considering the clojure community and a little too much "find the silver bullet" for my tastes. That said, I understand why a security focused person would want this and if my role was to be concerned about these things I would probably push in that direction as well. When taking a larger view however, this is a poor tradeoff IMO.
"I wouldn't want to give up the approach of composing a bunch of libraries to build concise, easy-to-reason-about web apps in Clojure--I think it helps with security in and of itself--but the community could really benefit from having more libraries which nail these things down and help developers build web apps more safely, with less work."
You nailed it, well said. I want a checklist and good libraries, not a framework that promises to handle things for you but quietly leaves a big hole open in that one part where you had to work around the framework to solve your problem (there is always at least one isn't there?).
You just can't avoid understanding these common attacks and how to mitigate them, at least at a surface level. You have to at least know where to apply the appropriate techniques, the implementation of that technique is left up to a well documented library written by experts.
"That said, I don't think it's all bad: you can build high quality, secure web apps in Clojure, it just requires you to really pay attention to security. And we should be doing that anyways, right?"
Not only should you be doing that, but if you aren't designing with security in mind then you are doing it wrong IMHO, "secure by default" framework or not. Either the people configuring the framework or the people writing the layers at the edges need to understand web security.
All that said, I understand the push in this direction. If you have a large team of junior devs then you want to limit their power. It's just not an approach that fits very well with Clojure. It's like Rich Hickey's attitude towards unbounded queues: there is no support for them, because you must be aware of this issue and hiding it causes bugs. There, now I've invoked Rich Hickey to justify my anti-framework bias ... so I must be right.
They struck me as both unrealistic considering the clojure community and a little too much "find the silver bullet" for my tastes. That said, I understand why a security focused person would want this and if my role was to be concerned about these things I would probably push in that direction as well. When taking a larger view however, this is a poor tradeoff IMO.
"I wouldn't want to give up the approach of composing a bunch of libraries to build concise, easy-to-reason-about web apps in Clojure--I think it helps with security in and of itself--but the community could really benefit from having more libraries which nail these things down and help developers build web apps more safely, with less work."
You nailed it, well said. I want a checklist and good libraries, not a framework that promises to handle things for you but quietly leaves a big hole open in that one part where you had to work around the framework to solve your problem (there is always at least one isn't there?).
You just can't avoid understanding these common attacks and how to mitigate them, at least at a surface level. You have to at least know where to apply the appropriate techniques, the implementation of that technique is left up to a well documented library written by experts.
"That said, I don't think it's all bad: you can build high quality, secure web apps in Clojure, it just requires you to really pay attention to security. And we should be doing that anyways, right?"
Not only should you be doing that, but if you aren't designing with security in mind then you are doing it wrong IMHO, "secure by default" framework or not. Either the people configuring the framework or the people writing the layers at the edges need to understand web security.
All that said, I understand the push in this direction. If you have a large team of junior devs then you want to limit their power. It's just not an approach that fits very well with Clojure. It's like Rich Hickey's attitude towards unbounded queues: there is no support for them, because you must be aware of this issue and hiding it causes bugs. There, now I've invoked Rich Hickey to justify my anti-framework bias ... so I must be right.
"just pick one templating language"
He has a point. Is one really that much better than another? Also, shouldn't it be possible to come up with a single library to do HTML escaping that has the suitably paranoid defaults, which is then used by all the templating languages? Is one templating language's code for that going to be so much revolutionarily better than another's?
"everyone use a (heavy, highly opinionated) framework", etc.
My understanding is that he pointed that out as a solution other environments have taken (Rails) but that there was also value in the composable way of doing things in Clojure. His actual position, I understood as, "there must be some way of unifying this for Clojure."
He has a point. Is one really that much better than another? Also, shouldn't it be possible to come up with a single library to do HTML escaping that has the suitably paranoid defaults, which is then used by all the templating languages? Is one templating language's code for that going to be so much revolutionarily better than another's?
"everyone use a (heavy, highly opinionated) framework", etc.
My understanding is that he pointed that out as a solution other environments have taken (Rails) but that there was also value in the composable way of doing things in Clojure. His actual position, I understood as, "there must be some way of unifying this for Clojure."
In regards to templates: yes - there are drastically different ways to do templating. Enlive style templates are plain HTML files and clojure transformation functions, hiccup style are clojure data structures that mimic html structure and render to html and there are the placeholder template languages involving intermixing a crippled language and HTML (Django templates, Ruby's ERB, etc).
It would be business suicide to demand all your web designers learn clojure edn syntax in order to write out HTML in hiccup. They have work to do and they are experts with their existing tools, including quite complex IDEs.
If the html is being generated programmatically by clojure devs then hiccup type formats are extremely efficient. String building techniques suck worse for HTML than they do for SQL.
By far the most common types of template languages are the placeholder type. People love them. I hate them, moustache/ctemplate style templates are an excellent attempt to remove logic from them but why not just go all the way towards an enlive style if you recognize this benefit? I'm not the right person to defend them but the general argument is that designers can learn the crippled DSL easily and they can implement the simple parts of the template logic themselves. Moustache style templates are also useful if you are templating something like plain text that doesn't have structure.
And yes, I absolutely think that the single library to do HTML escaping with extremely paranoid defaults is the right solution. But a security person is going to be nervous that developers will not use it every place they should and unless I'm remembering incorrectly the request was for this to be integrated into the templating library to remove that error path. I'm not even sure how that could work properly outside of the placeholder paradigm (it has been done well in Haskell using types to track escaped/unescaped segments if memory serves).
In regards to frameworks: I think you are right about this position, I just really very much disagree that there "must be some way of unifying this in clojure" outside of an opinionated framework. The problem is the lack of ability to enforce a specific dataflow / data transformations that can be audited for security and generally applied. Enforcing a specific dataflow is the definition of a framework IMHO.
I fully agree with his push for more and better web security related libraries (I'm tired of doing these things myself but don't have the security expertise to create general solutions that anyone should trust) but some of it struck me as "if we just limit the expressive power of clojure for web development we can mitigate a large swath of security issues". He's absolutely right, but I have found that tradeoff to be a very poor one to choose outside of the simplest of CRUD apps.
It would be business suicide to demand all your web designers learn clojure edn syntax in order to write out HTML in hiccup. They have work to do and they are experts with their existing tools, including quite complex IDEs.
If the html is being generated programmatically by clojure devs then hiccup type formats are extremely efficient. String building techniques suck worse for HTML than they do for SQL.
By far the most common types of template languages are the placeholder type. People love them. I hate them, moustache/ctemplate style templates are an excellent attempt to remove logic from them but why not just go all the way towards an enlive style if you recognize this benefit? I'm not the right person to defend them but the general argument is that designers can learn the crippled DSL easily and they can implement the simple parts of the template logic themselves. Moustache style templates are also useful if you are templating something like plain text that doesn't have structure.
And yes, I absolutely think that the single library to do HTML escaping with extremely paranoid defaults is the right solution. But a security person is going to be nervous that developers will not use it every place they should and unless I'm remembering incorrectly the request was for this to be integrated into the templating library to remove that error path. I'm not even sure how that could work properly outside of the placeholder paradigm (it has been done well in Haskell using types to track escaped/unescaped segments if memory serves).
In regards to frameworks: I think you are right about this position, I just really very much disagree that there "must be some way of unifying this in clojure" outside of an opinionated framework. The problem is the lack of ability to enforce a specific dataflow / data transformations that can be audited for security and generally applied. Enforcing a specific dataflow is the definition of a framework IMHO.
I fully agree with his push for more and better web security related libraries (I'm tired of doing these things myself but don't have the security expertise to create general solutions that anyone should trust) but some of it struck me as "if we just limit the expressive power of clojure for web development we can mitigate a large swath of security issues". He's absolutely right, but I have found that tradeoff to be a very poor one to choose outside of the simplest of CRUD apps.
"it has been done well in Haskell using types to track escaped/unescaped segments if memory serves"
Right, that's done in Hamlet (which was developed for Yesod, but is its own library).
http://hackage.haskell.org/package/hamlet
Right, that's done in Hamlet (which was developed for Yesod, but is its own library).
http://hackage.haskell.org/package/hamlet
Another important thing he points at is to not write you own incarnation of every library you find lacking but provide patches/support/whatever to existing ones. While it might help your CV to have just another template library (which will inevitably be lacking again) on github, the ecosystem profits more from a few well defined and long-term maintained choices.
"Yet Another Unmaintained Library" is the new useless CS master's thesis.
We need to give it special consideration. Web libraries could make an effort to do an internal audit and maybe post a disclaimer.
As a community we should highlight libraries with a focus on security as well. I think key is focusing on making this a community consideration early and not falling into the same pit of issues as other popular languages/frameworks which shall remain nameless for the purpose of this comment.
As a community we should highlight libraries with a focus on security as well. I think key is focusing on making this a community consideration early and not falling into the same pit of issues as other popular languages/frameworks which shall remain nameless for the purpose of this comment.
The title should be: Don't assume security is happening automatically under the hood.
Even if using a web framework which supports security out of the box, don't take it for granted either. For example: that's a common error developers make when using Spring Security. It's powerful and full-featured, and developers get security and everything in a breeze, but when doing things on the fly misconfiguration happens and vulnerabilities are out there in the open.
Even if using a web framework which supports security out of the box, don't take it for granted either. For example: that's a common error developers make when using Spring Security. It's powerful and full-featured, and developers get security and everything in a breeze, but when doing things on the fly misconfiguration happens and vulnerabilities are out there in the open.
I believe this warning could apply to any young ecosystem or micro-framework (e.g., bottle). Could web applications written in Go, or to a lesser extent, node.js, suffer from the same shortcomings?
After working so much in Django, I've been interested in smaller, leaner solutions, but then, as my feature list keeps growing, I find I miss all the bells and whistles provided by larger frameworks. I did not think of the security implication before reading this article.
After working so much in Django, I've been interested in smaller, leaner solutions, but then, as my feature list keeps growing, I find I miss all the bells and whistles provided by larger frameworks. I did not think of the security implication before reading this article.
> Could web applications written in Go, or to a lesser extent, node.js, suffer from the same shortcomings?
Yes, and they do. And there's no "lesser extent"; if I can't set up a fresh node app with the correct defaults for all of this stuff - if it's not safe out of the box, and I don't have to go out of my way to pick crappy settings - then it's just as vulnerable.
People made a lot of fun out of the Rails vulns that hit last year but they were the visible growing pains of having to mature a framework. It's not about the individual bits, it's about the ecosystem at large.
Yes, and they do. And there's no "lesser extent"; if I can't set up a fresh node app with the correct defaults for all of this stuff - if it's not safe out of the box, and I don't have to go out of my way to pick crappy settings - then it's just as vulnerable.
People made a lot of fun out of the Rails vulns that hit last year but they were the visible growing pains of having to mature a framework. It's not about the individual bits, it's about the ecosystem at large.
My dream someday is to write a secure-by-default web framework. I've been collecting a list of things it needs to do. It's not surprising that few-to-no frameworks get it even remotely right... there's a lot more to it than meets the eye. CSRF, for instance, is hard to get right. There's a lot of "CSRF" implementations out there, where the scare quotes are part of how I'm naming them. There's headers to avoid clickjacking, which you really ought to default to the strongest protection level and make the user explicitly loosen if they need it. There's "Content Security Policy" headers that ought to be configured. And there's a pervasive problem where the security model for resource access is just laughable... it's up to you to hopefully authenticate a user correctly, and hopefully, you authorize them (hopefully, your framework realizes those are two distinct concepts! but I gotta tell you, the odds are against you), and hopefully, resources don't default to being universally available, and hopefully, that code is all correct.... there's stuff you really ought to do to the querystring to prevent enumeration attacks that would be interesting, there's the whole matter of whether your template permits, encourages, or begs you to shoot your entire site full of cross-site scripting attacks unless you laboriously explain to the template system each and every time "no, I don't want an XSS here... no, I don't want an XSS here... no, I don't want an XSS here" (even after 15+ years of experience here, I still can not reliably work in a templating system like this, something always slips through)... session management is a great deal harder than it looks... framework users should probably actually not be given access to the user's cookie in any direct form because that's begging for problems (darned near anything you can do to a cookie is insecure), to say nothing of how they ought to default to no HTTP and Secure... it's a nightmare out there.
Though we don't hit every one of your points, the Lift framework (http://liftweb.net/) hits a lot of them, and aims to be secure by default as much as possible. XSS is difficult by default because everything lives in a reified XML tree more or less until it goes down the pipe. SiteMap, which is the way that you do routing, disables access to anything that isn't explicitly declared in the site map. CSRF is based on randomized ids for each field and form, tied to this session, this page, and a specific callback function on the server. Content-Security-Policy isn't configured by default at the moment, though that's a good idea to add. X-Frame-Options is set by default. A bit more at http://seventhings.liftweb.net/security .
It took me about 5 minutes to realize Lift is for Scala. It doesn't say that anywhere on the website that I could tell.
Yeah… The site needs work. So does some of the intro documentation (there are a few good books on it, but I mean on-site intro documentation). We're working on these things, and hope to have something awesome to show in the next few months.
My dream someday is to write a secure-by-default web framework.
Devil's Advocacy: How about a front-facing Web Server/WebApp Firewall/language hosting environment? Build something language neutral that can be trivially set up with no more than 3 command-line options and the name of the language/web-framework and you've got the secure defaults out of the box?
Devil's Advocacy: How about a front-facing Web Server/WebApp Firewall/language hosting environment? Build something language neutral that can be trivially set up with no more than 3 command-line options and the name of the language/web-framework and you've got the secure defaults out of the box?
You could get some mileage that way for some of the more trivial stuff, but stuff like user authorization (not authentication, authorization) tends to call for deeper integration into what is probably a framework. (Creating a you-call-me library would probably make it too easy to accidentally bypass the security, vs. an I-call-you framework that is guaranteed to go through the secure code path before and after your code.)
Still, if there was a standard for middleware in general you probably could at least get all the headers right and provide some sort of safe, general CSRF, session, and authentication (not authorization) framework that way. Unfortunately, the easiest places to stick a generalized middleware, which is probably an nginx module, also tend to lack any sort of backend storage, which is a problem.
Whether or not you could sell anyone on the benefits of that would be an interesting question. Getting any sort of XSS protection would involve essentially putting a full Web Application Firewall in front, for instance, which would be infeasible with any reasonable effort, and I'm not sure anyone's going to look twice at a "web security platform" that does little to nothing (effective) against XSS.
Still, if there was a standard for middleware in general you probably could at least get all the headers right and provide some sort of safe, general CSRF, session, and authentication (not authorization) framework that way. Unfortunately, the easiest places to stick a generalized middleware, which is probably an nginx module, also tend to lack any sort of backend storage, which is a problem.
Whether or not you could sell anyone on the benefits of that would be an interesting question. Getting any sort of XSS protection would involve essentially putting a full Web Application Firewall in front, for instance, which would be infeasible with any reasonable effort, and I'm not sure anyone's going to look twice at a "web security platform" that does little to nothing (effective) against XSS.
I've looked at a lot of Node, Sinatra, and Compojure web apps and the vast majority of them are vulnerable to a variety of attacks. CSRF protection in particular is almost never handled and when it is it's often not done properly.
CSRF protection is handled by Django, for example, by default - an example of something you'd be giving up if you decided that Django is too "big"; now you need to figure this out yourself and find a library that provides it or program it yourself and hope you get it right.
I would want to turn the question around and (genuinely, not rhetorically) ask: which ecosystems right now do have a good security baseline? Am I wrong when I guess Java/.Net/Ruby and possibly more recent PHP frameworks?
ASP.NET MVC 4 and later is pretty decent, Rails is getting better but recently had a large series of problems, and django is one of the most secure by default web frameworks that I've ever used. Django really gets a lot of things (sessions, CSRF, XSS, etc..) right out of the box. Grails is pretty decent but I don't have a lot of in depth Java framework experience but it seems to vary based on the framework.
I will leave this here
http://www.javalobby.org/articles/acegisecurity/part1.jsp
I felt some relief after watching the presentation. I thought I was just a whiny newbie who was spoiled by Python/AppEngine (although I guess the spoiled part is true). Turns out it is a real issue with the Clojure ecosystem.
Another talk from the conference I'm glad was presented is "How Clojure Works"[1]. Towards the end, he shows the negative effects on tooling caused by Clojure's dynamic + hosted nature. Clojure's stacktraces are a nightmare. And the painful startup time kind of negates the excitement a newcomer might have towards REPL-oriented development.
Hopefully it won't be too long until things improve.
[1]: https://www.youtube.com/watch?v=8NUI07y1SlQ&list=PLZdCLR02gr...
Another talk from the conference I'm glad was presented is "How Clojure Works"[1]. Towards the end, he shows the negative effects on tooling caused by Clojure's dynamic + hosted nature. Clojure's stacktraces are a nightmare. And the painful startup time kind of negates the excitement a newcomer might have towards REPL-oriented development.
Hopefully it won't be too long until things improve.
[1]: https://www.youtube.com/watch?v=8NUI07y1SlQ&list=PLZdCLR02gr...
I felt some relief after watching the presentation. I thought I was just a whiny newbie who was spoiled by Python/AppEngine (although I guess the spoiled part is true).
I would hazard a guess that some of this reflects on the Clojure community. (Not that this is somehow inherent or uncorrectable.)
I would hazard a guess that some of this reflects on the Clojure community. (Not that this is somehow inherent or uncorrectable.)
Oh I thought to myself that I was a whiny newbie. The community's (at least, irc and subreddit) pretty cool.
It's always better to have more people reviewing the security of an application, even if they seem like a whiny newbie.
"the painful startup time kind of negates the excitement a newcomer might have towards REPL-oriented development."
But you just need to start up once and then work all day in the REPL. Unless you mean that the JVM startup means that Clojure is not a good choice for writing utility scripts and such.
EDIT: As pointed out to me below, and in the excellent talk linked above, it's much more the Clojure bootstrapping time, which almost dwarfs the JVM startup time.
But you just need to start up once and then work all day in the REPL. Unless you mean that the JVM startup means that Clojure is not a good choice for writing utility scripts and such.
EDIT: As pointed out to me below, and in the excellent talk linked above, it's much more the Clojure bootstrapping time, which almost dwarfs the JVM startup time.
The startup time that makes Clojure unsuitable for utility scripts and such is really Clojure's, not the JVM's! JVM start up time is actually quite small these days. See for example, http://nicholaskariniemi.github.io/2014/03/19/solving-clojur...
Heh. Back in the day, Smalltalk had a reputation for being too slow to write command-line apps and utility scripts. There were even bits of wonderful tech developed that could produce Smalltalk images as small as 45k!
But even more surprising: You could take the default VisualWorks, tell it to skip the herald sound and the splash screen and shut a few other things off, and it would load its 12MB default image faster than Perl 5 could start up with a 768k memory footprint.
But even more surprising: You could take the default VisualWorks, tell it to skip the herald sound and the splash screen and shut a few other things off, and it would load its 12MB default image faster than Perl 5 could start up with a 768k memory footprint.
Web development is very particular because of security. You can write, say, a console newsread with any programming language that has an XML parser and a curses binding. But if you want to write a web application that will actually see production, you need a web framework that has been sufficiently battle tested and audited by security-competent people.
Which is a shame, because I'd really like to be able to use Common Lisp for web development. But as much as I love the language, you have to be reasonable.
Which is a shame, because I'd really like to be able to use Common Lisp for web development. But as much as I love the language, you have to be reasonable.
> you need a web framework that has been sufficiently battle tested and audited by security-competent people
Or alternatively, you just need to be careful with keeping things simple and straightforward. IMHO frameworks only hide problems somewhere else beneath layers of complexity. Assuming a fixed bug-rate, the more code there is, the more bugs that will be present (and whether they get discovered is another issue.)
Or alternatively, you just need to be careful with keeping things simple and straightforward. IMHO frameworks only hide problems somewhere else beneath layers of complexity. Assuming a fixed bug-rate, the more code there is, the more bugs that will be present (and whether they get discovered is another issue.)
The argument is that making things too simple is what kills you. The defaults are not secure and you need to take steps to make sure you are secure. This is one area where frameworks are huge plus. Good ones ship with security conscious defaults.
This isn't about bugs in the sense of "code that is incorrectly written for what it was meant to do." It's more about bugs in the sense of "code that was written correctly, but which does not do things it should do because you either didn't think of or didn't care about them when you were writing it." It is easier and often simpler to allow vulnerabilities than it is to prevent them.
I've used Weblocks, Slava Akhmechet's CL framework, and I think it's pretty good. The database interface is abstracted so you never have to cons up SQL directly. Session handling and CSRF protection are built in. I did find an XSS bug or two in it, which I've fixed.
You still have to pay attention to XSS when writing your app -- there's no way around that -- but I don't think there are a lot of holes in the framework itself, and it makes it relatively easy to write a secure app.
I can't offer you a guarantee, of course, but I can tell you that I'm less nervous about Weblocks than I would be about Rails.
EDITED to add: Weblocks does have a pretty substantial learning curve, but once you know how to use it, it can make for very high productivity.
You still have to pay attention to XSS when writing your app -- there's no way around that -- but I don't think there are a lot of holes in the framework itself, and it makes it relatively easy to write a secure app.
I can't offer you a guarantee, of course, but I can tell you that I'm less nervous about Weblocks than I would be about Rails.
EDITED to add: Weblocks does have a pretty substantial learning curve, but once you know how to use it, it can make for very high productivity.
You news reader example is a poor one. There have been a ton of vulnerabilities in XML parsing libraries over the years. Of course the attack surface is different (I need to engineer an attack that makes you open my vulnerable XML stream versus bang away on your web app), but the vulnerabilities are still there and as a programmer you need to be security conscious on every platform.
I've spent the last 3 years developing in Clojure and found I was quite happy not having an all encompassing framework. It forced me to learn all these issues and understand them well enough to be able to customize each and align them with a products design.
Sure it's more work, but in my mind, it's worth it.
Sure it's more work, but in my mind, it's worth it.
> It forced me to learn all these issues and understand them well enough to be able to customize each and align them with a products design.
This is a standard developer response - I recall when Sinatra was a fresh kid on the block, and the same arguments people used against Rails - but just because something is more fun doesn't mean you're going to get all these details right.
The attack surface in modern web apps can be such a large search space that the majority of people will never get it right. The point behind frameworks is that you can engage in efficiencies in scale when it comes to sensible defaults; even the most well seasoned and security paranoid developer can drown in the cornucopia of details that all have to be done right.
This is a standard developer response - I recall when Sinatra was a fresh kid on the block, and the same arguments people used against Rails - but just because something is more fun doesn't mean you're going to get all these details right.
The attack surface in modern web apps can be such a large search space that the majority of people will never get it right. The point behind frameworks is that you can engage in efficiencies in scale when it comes to sensible defaults; even the most well seasoned and security paranoid developer can drown in the cornucopia of details that all have to be done right.
> This is a standard developer response
vs. a what? I've worn many hats in my day, the least of which has been a developer hat.
There's equal risk in relying on a code base incorporating features with underlying concepts that you or your team does not understand.
I'm all for having a framework, where you can review the code and be comfortable with what you're relying on, but in the absence of having a good one, well then build one. The benefits Clojure brings to the table more than make up for the work in either case.
There's just no excuse for not knowing the concepts and not knowing if your product needs them.
vs. a what? I've worn many hats in my day, the least of which has been a developer hat.
There's equal risk in relying on a code base incorporating features with underlying concepts that you or your team does not understand.
I'm all for having a framework, where you can review the code and be comfortable with what you're relying on, but in the absence of having a good one, well then build one. The benefits Clojure brings to the table more than make up for the work in either case.
There's just no excuse for not knowing the concepts and not knowing if your product needs them.
>The benefits Clojure brings to the table more than make up for the work in either case.
No one's saying "stop using clojure". The argument here is "build a clojure framework lots of people use".
>There's equal risk in relying on a code base incorporating features with underlying concepts that you or your team does not understand.
You do this all the time. I mean, I understand on a high level what my computer does but I can't describe to you the exact tree balancing algorithm my filesystem uses, how my CPU performs cache invalidation and branch prediction or even how exactly the JVM works.
Computers are too complex to not rely on underlying concepts you or your team doesn't understand.
The point is, within just the context of a web developer developing a web application there is too much stuff to know to always get it right. We need to reduce the number of things people have to think about, or else we'll always miss something.
No one's saying "stop using clojure". The argument here is "build a clojure framework lots of people use".
>There's equal risk in relying on a code base incorporating features with underlying concepts that you or your team does not understand.
You do this all the time. I mean, I understand on a high level what my computer does but I can't describe to you the exact tree balancing algorithm my filesystem uses, how my CPU performs cache invalidation and branch prediction or even how exactly the JVM works.
Computers are too complex to not rely on underlying concepts you or your team doesn't understand.
The point is, within just the context of a web developer developing a web application there is too much stuff to know to always get it right. We need to reduce the number of things people have to think about, or else we'll always miss something.
> The argument here is "build a clojure framework lots of people use.
The argument I am addressing is that people are deploying products with poor security because a great, all encompassing, framework does not exist. There's no excuse for doing that.
This is not about having to understand cryptography, it's about knowing when and where you need to use it and making sure it's included when you do. The issues outlined in the video are trivial.
The argument I am addressing is that people are deploying products with poor security because a great, all encompassing, framework does not exist. There's no excuse for doing that.
This is not about having to understand cryptography, it's about knowing when and where you need to use it and making sure it's included when you do. The issues outlined in the video are trivial.
The excuse is precisely that there isn't a great, all encompassing framework.
Setting up a scenario where everyone has to get everything right all of the time is setting everyone up for a lot of heartbreak.
Having bad password management and opening everyone up to CSRF and XSS is not "trivial". They're gotchas, and looking out for gotchas is a waste of everyone's time.
Setting up a scenario where everyone has to get everything right all of the time is setting everyone up for a lot of heartbreak.
Having bad password management and opening everyone up to CSRF and XSS is not "trivial". They're gotchas, and looking out for gotchas is a waste of everyone's time.
When I make mistakes, which I often do, I don't go around thinking oh, darn, they should have known that for me. That goes for using a bad framework or lack thereof.
As I said, I'm all for having a great framework, but I would not excuse the mistakes being made here, and I'm simply saying that I believe that my products and my knowledge level benefited by not having one.
> Having bad password management and opening everyone up to CSRF and XSS is not "trivial".
I'm not suggesting everyone should need to be aware of these issues, I'm saying someone on your team should be. And if you're a team of one, be prepared to make mistakes (which will not be limited to the issues outlined here).
As I said, I'm all for having a great framework, but I would not excuse the mistakes being made here, and I'm simply saying that I believe that my products and my knowledge level benefited by not having one.
> Having bad password management and opening everyone up to CSRF and XSS is not "trivial".
I'm not suggesting everyone should need to be aware of these issues, I'm saying someone on your team should be. And if you're a team of one, be prepared to make mistakes (which will not be limited to the issues outlined here).
>When I make mistakes, which I often do, I don't go around thinking oh, darn, they should have known that for me.
Well, you should! These are mind numbing details that are best taken care of by other people. I am too concerned with building a good product to want to spend too much time going over mind numbing details.
The argument from "I become a more well rounded developer" is valid, but these should be things you learn once and then stop worrying about forever. There's a lot of magic in Rails, but it all goes away given enough time. Better abstractions are our bread and butter and we'll get nothing done if we continue to shy away from them.
Well, you should! These are mind numbing details that are best taken care of by other people. I am too concerned with building a good product to want to spend too much time going over mind numbing details.
The argument from "I become a more well rounded developer" is valid, but these should be things you learn once and then stop worrying about forever. There's a lot of magic in Rails, but it all goes away given enough time. Better abstractions are our bread and butter and we'll get nothing done if we continue to shy away from them.
As I said, I'm all for having a great framework.
Unfortunately, with security, you have to "get everything right all of the time" because the attacker only has to get their attack right once.
I worded it unfortunately, but we're getting at the same thing.
What I mean to say is the easier it becomes for everyone to get everything right all of the time the more likely it is to occur.
Expecting everyone to individually know and understand every single gotcha is unrealistic compared to distributing the work load across a framework.
What I mean to say is the easier it becomes for everyone to get everything right all of the time the more likely it is to occur.
Expecting everyone to individually know and understand every single gotcha is unrealistic compared to distributing the work load across a framework.
> "This is a standard developer response - I recall when Sinatra was a fresh kid on the block, and the same arguments people used against Rails"
And the more time passes, the more problems with Rails are being discovered that have to be altered in a non-BC way.
A library makes no assumptions. It's just there and you decide how to use it. In the right hands it's more powerful than a rigid framework.
A framework that tries to teach you how to do things is by definition a pair of training wheels. Not the best setup for competing in Tour De France.
The solution to incompetent developers isn't a bigger, more rigid framework. It's education and training.
And the more time passes, the more problems with Rails are being discovered that have to be altered in a non-BC way.
A library makes no assumptions. It's just there and you decide how to use it. In the right hands it's more powerful than a rigid framework.
A framework that tries to teach you how to do things is by definition a pair of training wheels. Not the best setup for competing in Tour De France.
The solution to incompetent developers isn't a bigger, more rigid framework. It's education and training.
The solution to incompetent developers isn't a bigger, more rigid framework. It's education and training.
In aerospace, it's understood that better training can be part of the solution, but often better instruments and interfaces and well designed automated tools can be lifesaving as well.
In aerospace, it's understood that better training can be part of the solution, but often better instruments and interfaces and well designed automated tools can be lifesaving as well.
Of course, and in software development we call these instruments, interfaces and well designed automated tools libraries.
A framework in aerospace would be more akin to a pre-built airplane with a bunch of holes where you put seats, in-flight entertainment consoles and other auxiliary non-critical equipment (because remember, airplanes are scary and you were told you can't do it yourself). Then you paint it and stick your logo on it.
Which is all fine, if it does the job for you, but you're not designing the airplane, anymore than microwaving TV lunch is you cooking. And just like warming up TV lunch doesn't make your job title a "chef", I don't believe people relying on frameworks to do the hard thinking for them deserve the title "architect" or even "senior developer". Which, again, is all fine, unless you or your boss are kidding yourselves about it.
And then one day the military, say, realize they now have a bunch of useless "military jets" built on top of a "passenger airplane framework". And NASA's space mission based on that framework isn't going too well, either. Whoops. Turns out frameworks aren't one-size-fits-all.
In a nutshell, apps have to be architected by competent architects, using simple, modular, does-one-thing-well libraries. These people know how to achieve scalability and security taking the particular needs of the app at hand.
The presenter in that video said that making assumptions is dangerous. There's no greater assumption than the fact a framework is right for your app, without understanding what this framework does under the hood. And there's no type of component making more assumptions for your app than a framework. Rails calls it "opinionated". I call it: you're the framework's slave, not the other way around.
There's no way around it. I know when Rails came out the hype was so huge, that everyone believed they and their dog can write big complex web apps. Nope. The only result of the framework mentality is a bunch of people making beginner mistakes and getting hacked all over the web.
A framework in aerospace would be more akin to a pre-built airplane with a bunch of holes where you put seats, in-flight entertainment consoles and other auxiliary non-critical equipment (because remember, airplanes are scary and you were told you can't do it yourself). Then you paint it and stick your logo on it.
Which is all fine, if it does the job for you, but you're not designing the airplane, anymore than microwaving TV lunch is you cooking. And just like warming up TV lunch doesn't make your job title a "chef", I don't believe people relying on frameworks to do the hard thinking for them deserve the title "architect" or even "senior developer". Which, again, is all fine, unless you or your boss are kidding yourselves about it.
And then one day the military, say, realize they now have a bunch of useless "military jets" built on top of a "passenger airplane framework". And NASA's space mission based on that framework isn't going too well, either. Whoops. Turns out frameworks aren't one-size-fits-all.
In a nutshell, apps have to be architected by competent architects, using simple, modular, does-one-thing-well libraries. These people know how to achieve scalability and security taking the particular needs of the app at hand.
The presenter in that video said that making assumptions is dangerous. There's no greater assumption than the fact a framework is right for your app, without understanding what this framework does under the hood. And there's no type of component making more assumptions for your app than a framework. Rails calls it "opinionated". I call it: you're the framework's slave, not the other way around.
There's no way around it. I know when Rails came out the hype was so huge, that everyone believed they and their dog can write big complex web apps. Nope. The only result of the framework mentality is a bunch of people making beginner mistakes and getting hacked all over the web.
And then one day the military, say, realize they now have a bunch of useless "military jets" built on top of a "passenger airplane framework".
The He 111 started out as a civilian plane, then became a bomber. But to be fair, it was always a bomber masquerading as a civilian plane.
http://en.wikipedia.org/wiki/Heinkel_He_111
Which is all fine, if it does the job for you, but you're not designing the airplane, anymore than microwaving TV lunch is cooking.
I don't think this is a very good analogy at all, unless it comes to something like App Engine or Heroku, and I'm not sure even then.
I know when Rails came out the hype was so huge, that everyone believed they and their dog can write big complex web apps. Nope. The only result of that is a bunch of people making beginner mistakes and getting "pwned" all over the web.
The smart move would've been to see this coming, then intervene, thus cementing that big huge groundswell of mindshare with better informed members of the community as a result. The thing is, we programmers are more apt to act like old-time barnstormers and solemnly shake our heads at the pilot who stalled and crashed, than to invent stick shakers and automated stall warnings.
The He 111 started out as a civilian plane, then became a bomber. But to be fair, it was always a bomber masquerading as a civilian plane.
http://en.wikipedia.org/wiki/Heinkel_He_111
Which is all fine, if it does the job for you, but you're not designing the airplane, anymore than microwaving TV lunch is cooking.
I don't think this is a very good analogy at all, unless it comes to something like App Engine or Heroku, and I'm not sure even then.
I know when Rails came out the hype was so huge, that everyone believed they and their dog can write big complex web apps. Nope. The only result of that is a bunch of people making beginner mistakes and getting "pwned" all over the web.
The smart move would've been to see this coming, then intervene, thus cementing that big huge groundswell of mindshare with better informed members of the community as a result. The thing is, we programmers are more apt to act like old-time barnstormers and solemnly shake our heads at the pilot who stalled and crashed, than to invent stick shakers and automated stall warnings.
Web Application is one of the places where Clojure's java compatibility falls short. I have a ton of time invested in Spring + ACEGI security and it just works. I have been exploring on how to develop servlets, which can use servlet session, in Clojure and they all look very ugly compared to normal Compojure based apps. I am coming to the opinion that maybe its not ready for J2EE web apps , which is a shame as J2EE has a very rich ecosystem.
All non-mainstreem web frameworks are very immature in terms of web security. No CSRF/XSS/SQLi projections built in, don't even say about security headers. If you use an unpopular framework you should do all protections yourself.
Apparently you don't know yesod[1].
[1] http://yesodweb.com
[1] http://yesodweb.com
they can claim to be secure, but nobody spent enough time to audit them. Too unpopular
AFAIK, the Flask-WTF extension for Flask does CSFRF for forms. Flask also protects from XSS per default.
SQL injection is not a problem of web frameworks, because SQL should never be a part of the web layer of an application in the first place.
Virtually all web frameworks encourage SQL. So it is a problem. Making a private api to access the database can add a lot of needless complexity.
That private API is called a service layer, and without it, your code devolves into copy/paste spaghetti.
We live in an age when having a complementary iOS/Android app for your site is not the exception anymore. Without a service layer you'll have coupled your services with your web pages. Connecting your server with your apps will be quite painful. You don't want that.
Done right, your service layers keeps your code simple, secure, and easy to debug, even if it's web-only. I'd argue that anything else is just due to lack of experience.
A service layer also makes the work on a project more parallelizable in terms of number of developers that can work together on it, each focusing on their part of the puzzle, without breaking each other's code all the time.
We live in an age when having a complementary iOS/Android app for your site is not the exception anymore. Without a service layer you'll have coupled your services with your web pages. Connecting your server with your apps will be quite painful. You don't want that.
Done right, your service layers keeps your code simple, secure, and easy to debug, even if it's web-only. I'd argue that anything else is just due to lack of experience.
A service layer also makes the work on a project more parallelizable in terms of number of developers that can work together on it, each focusing on their part of the puzzle, without breaking each other's code all the time.
But then your service layer has potential SQLi issues. Moving the problem doesn't make it go away. Many of the clojure SQL libraries solve this by using prepared statements by default. This is not a hard problem.
If it's "not a hard problem" - and I agree it isn't, and it's solved by Clojure's SQL libraries, why should the web framework deal with it in any way?
And moving the problem actually helps, when the original place was the wrong place to solve the problem.
Security is one problem when you write queries all over the place. The other two problems are data integrity, and maintenance.
In a service layer you have exactly three concerns:
1) Validate abstract input (and permissions of the caller).
2) Perform the transaction.
3) Return abstract output (and/or errors).
No routers, no controllers, no views, no templates, no CSRF, no XSS, no HTML escaping, no GET, no POST, no nothing.
Just input, transaction, output. Pure data. Pure business logic.
And suddenly things that seemed hard to get right, or things you had to repeat all over the place in your code, get done simply, and just once. And all your web code calls the service layer.
All your iOS and Android app code - also calls the same service layer.
And they're both secure because you need to get the service secure just once - then all interfaces (web, mobile, desktop, API) use the same service.
And when you don't isolate the service, you'll be running SQL all over the place, and use every framework's souped up solutions to avoid SQLi.
Which one do you feel is better?
And moving the problem actually helps, when the original place was the wrong place to solve the problem.
Security is one problem when you write queries all over the place. The other two problems are data integrity, and maintenance.
In a service layer you have exactly three concerns:
1) Validate abstract input (and permissions of the caller).
2) Perform the transaction.
3) Return abstract output (and/or errors).
No routers, no controllers, no views, no templates, no CSRF, no XSS, no HTML escaping, no GET, no POST, no nothing.
Just input, transaction, output. Pure data. Pure business logic.
And suddenly things that seemed hard to get right, or things you had to repeat all over the place in your code, get done simply, and just once. And all your web code calls the service layer.
All your iOS and Android app code - also calls the same service layer.
And they're both secure because you need to get the service secure just once - then all interfaces (web, mobile, desktop, API) use the same service.
And when you don't isolate the service, you'll be running SQL all over the place, and use every framework's souped up solutions to avoid SQLi.
Which one do you feel is better?
I've seen SQLi blindly passed from frontend code to a backend service API and then executed. The indirection did help mask the issue so there was no suggestion the attack succeeded or way that it could actually be exploited, but it's perfectly possible to what you describe without actually solving the sanitization problem.
SOA is great for scalability, but not really that much of a boon for security (especially since you now have a problem of authentication between your frontend service and your backend service which is often ignored since it's behind the firewall and nobody is going to be snooping.) (Other than the NSA, GHCQ, and China.)
SOA is great for scalability, but not really that much of a boon for security (especially since you now have a problem of authentication between your frontend service and your backend service which is often ignored since it's behind the firewall and nobody is going to be snooping.) (Other than the NSA, GHCQ, and China.)
You can always shoot yourself in the foot even with a service layer.
But in a service layer you need to get it right once. And therefore if you don't, you have to fix it just once.
And coupling your business logic with your frontend layer suggests spaghetti code and violates DRY, because you typically have many frontends, but one app state.
Security is about focus. If the service coders can focus on the service being secure & fast, frontend guys can focus on the frontend being usable.
Otherwise you're asking everyone to think about everything, and human attention span, memory and skill sets are limited, and this does affect security.
But in a service layer you need to get it right once. And therefore if you don't, you have to fix it just once.
And coupling your business logic with your frontend layer suggests spaghetti code and violates DRY, because you typically have many frontends, but one app state.
Security is about focus. If the service coders can focus on the service being secure & fast, frontend guys can focus on the frontend being usable.
Otherwise you're asking everyone to think about everything, and human attention span, memory and skill sets are limited, and this does affect security.
It shouldn't be but that's not always possible.
It is always possible. Name one case where it isn't, and I'll tell you how it's possible :)
Web frameworks should handle input (routing, request fields, url query) and output (templates). That's it.
Web frameworks should handle input (routing, request fields, url query) and output (templates). That's it.
OK, Django ORM, conditional aggregates - these were not possible last time I checked. You can do it at the application level, but say you want to keep the performance up, you need to drop into SQL.
When I think of other developers like me that are building projects solo on top of Express/Sinatra/Compojure, the most helpful resource would be a simple checklist.
For example, the CSRF section could briefly explain what cross-site request forgery is, `<img src="http://example.com/logout">`, and recommend some approaches and good practices.
Code examples could be provided for various languages/frameworks each in its own tab. The Clojure tab could demonstrate https://github.com/weavejester/ring-anti-forgery and a `anti-forgery-form-to` macro that wraps `hiccup.form/form-to` and inserts `(ring.util.anti-forgery/anti-forgery-field)` into the form.
The XSS section could remind us to ensure that our templates are escaping user input. The Clojure tab could chart the safe vs unsafe templating libs. (Hiccup is unsafe)
The Sessions section could remind us to sign our cookies so that someone can't just change a cookie from {:user-id 42} to {:user-id 1}. And that our never-expiring authentication cookie means that anyone with that cookie can log in as that user forever. It could explain an overview of the main solutions and their drawbacks like how to back sessions by a database or set an expiration within the signed cookie.
Just like convos about crypto, discussions about security too frequently turn into some abstinence-only pamphlet.
For example, the CSRF section could briefly explain what cross-site request forgery is, `<img src="http://example.com/logout">`, and recommend some approaches and good practices.
Code examples could be provided for various languages/frameworks each in its own tab. The Clojure tab could demonstrate https://github.com/weavejester/ring-anti-forgery and a `anti-forgery-form-to` macro that wraps `hiccup.form/form-to` and inserts `(ring.util.anti-forgery/anti-forgery-field)` into the form.
The XSS section could remind us to ensure that our templates are escaping user input. The Clojure tab could chart the safe vs unsafe templating libs. (Hiccup is unsafe)
The Sessions section could remind us to sign our cookies so that someone can't just change a cookie from {:user-id 42} to {:user-id 1}. And that our never-expiring authentication cookie means that anyone with that cookie can log in as that user forever. It could explain an overview of the main solutions and their drawbacks like how to back sessions by a database or set an expiration within the signed cookie.
Just like convos about crypto, discussions about security too frequently turn into some abstinence-only pamphlet.
Just like convos about crypto, discussions about security too frequently turn into some abstinence-only pamphlet.
As opposed to a "Purity Test" as a checklist?
As opposed to a "Purity Test" as a checklist?
[deleted]
This presentation echoes a lot of what we've seen. For what it's worth, we scan a lot of Clojure apps and find a lot of the types of vulnerabilities Aaron/John are talking about. Feel free to run a free scan at https://www.tinfoilsecurity.com if you want to try it out. :) Would love feedback.
So what are the secure by default web frameworks apart Django and Rails ?
For something fairly modern, you might want to have a look at web2py. It goes to quite great lengths to balance sane-out-of-the-box and developer productivity:
http://www.web2py.com/
http://www.web2py.com/
Scala Lift
Nothing is secure by default, even Django and Rails. You can always shoot yourself in the foot, if you don't know the basics about security.
This is why the solution isn't relying on your framework, but on your competence.
This is why the solution isn't relying on your framework, but on your competence.
FTA:
> and session cookies don't default to http-only flags (allowing XSS attacks to steal them).
Can someone explain that to me?
> and session cookies don't default to http-only flags (allowing XSS attacks to steal them).
Can someone explain that to me?
Setting this flag on a cookie means it can't be accessed from JavaScript, so a malicious script won't be able to determine your session (for instance).
Take a look here: https://www.owasp.org/index.php/HttpOnly
Perhaps a better title would be "Complete beginner web apps in Clojure less secure than complete beginner web apps in other environments." (EDIT: Isn't security a perfect example of a place where (good tech + good community) > (miracle tech + bad community)?)
This brings to mind an incident, about which I'll say: While I do find that the community is friendly and a source of satisfying discussion in person, the Clojure community online is notably less beginner friendly than other language communities. (tl;dr - Attempts to "help" newbs that are really opportunities to be a language weenie considered harmful.)
I find some parallels with the Smalltalk community, which had a reputation for an "ivory tower" attitude and a "Not Invented Here" snobbishness. When I was a beginner in Smalltalk, a coworker of mine had written a nice but arbitrary little DSL specification language using Array literals and symbols. So I made some sort of comment about the syntax she'd created, and which prompted a knee-jerk reaction about how Smalltalk had a small non-ALGOL syntax -- as if I were totally unfamiliar with Smalltalk syntax.
I've had experiences like this asking about the arbitrary function definition-like syntax of Compojure web application routes. Those look somewhat like function definitions, but they're really macros. I had tried for some time to Google the answer, but no one had ever posted about the exact code to do what I wanted. (Which wasn't at all abstruse. Something like getting access to the entire request map while modifying the returned session.) Well, it turned out that all I needed to know was to not refer to the request as
Just what kind of community treats with newcomers by assuming that the newcomers are lying and then telling them to go read Lisp macros? Also, this is only the most easily related situation I've encountered while interacting with the Clojure community. There have been others.
No non-algol patterned language has ever gained more than a niche acceptance. If Clojure doesn't want to commit long term mindshare suicide, some attention needs to be paid to avoiding inadvertent hostility to newcomers. (This includes unintentionally being a bad security-defaults minefield.) Smalltalkers had the same "read the source" knee-jerk. In the end, it didn't work out so well. Care needs to be taken to understand exactly what the newbs are asking, because they know the least about how to formulate the question. Also, if you find yourself running a simplistic "pattern match" head-daemon eagerly looking for opportunities to bedazzle the newb with your wonderful, superior, and different way of doing things, just stop. You are giving your community a "weenie" reputation and your mental energies are better used elsewhere.
This brings to mind an incident, about which I'll say: While I do find that the community is friendly and a source of satisfying discussion in person, the Clojure community online is notably less beginner friendly than other language communities. (tl;dr - Attempts to "help" newbs that are really opportunities to be a language weenie considered harmful.)
I find some parallels with the Smalltalk community, which had a reputation for an "ivory tower" attitude and a "Not Invented Here" snobbishness. When I was a beginner in Smalltalk, a coworker of mine had written a nice but arbitrary little DSL specification language using Array literals and symbols. So I made some sort of comment about the syntax she'd created, and which prompted a knee-jerk reaction about how Smalltalk had a small non-ALGOL syntax -- as if I were totally unfamiliar with Smalltalk syntax.
I've had experiences like this asking about the arbitrary function definition-like syntax of Compojure web application routes. Those look somewhat like function definitions, but they're really macros. I had tried for some time to Google the answer, but no one had ever posted about the exact code to do what I wanted. (Which wasn't at all abstruse. Something like getting access to the entire request map while modifying the returned session.) Well, it turned out that all I needed to know was to not refer to the request as
[request]
request
Just what kind of community treats with newcomers by assuming that the newcomers are lying and then telling them to go read Lisp macros? Also, this is only the most easily related situation I've encountered while interacting with the Clojure community. There have been others.
No non-algol patterned language has ever gained more than a niche acceptance. If Clojure doesn't want to commit long term mindshare suicide, some attention needs to be paid to avoiding inadvertent hostility to newcomers. (This includes unintentionally being a bad security-defaults minefield.) Smalltalkers had the same "read the source" knee-jerk. In the end, it didn't work out so well. Care needs to be taken to understand exactly what the newbs are asking, because they know the least about how to formulate the question. Also, if you find yourself running a simplistic "pattern match" head-daemon eagerly looking for opportunities to bedazzle the newb with your wonderful, superior, and different way of doing things, just stop. You are giving your community a "weenie" reputation and your mental energies are better used elsewhere.
For what it's worth, I regularly browse the IRC logs to see if anything interesting has come up, and the only time I've ever seen any amount of conflict was when you were involved - and not the incident you're describing.
They seem generally pretty helpful to newbies, from my own selfish perspective to a fault, since I have to skip past the dozens of "how do I add to arrays" with clojurebot responses.
It might be in the way you're asking.
They seem generally pretty helpful to newbies, from my own selfish perspective to a fault, since I have to skip past the dozens of "how do I add to arrays" with clojurebot responses.
It might be in the way you're asking.
They seem generally pretty helpful to newbies, from my own selfish perspective to a fault, since I have to skip past the dozens of "how do I add to arrays" with clojurebot responses.
Lots of them seem to be stuck in "quick pattern match" mode, so in the other occasions they were "pattern matching" on my asking about a much simpler scenario than I was actually asking about -- then taking my trying to inform them their assumptions were wrong to conclude something like "he doesn't understand threads."
It might be in the way you're asking.
I say as much in my comment above -- I don't know enough about how to ask the questions in the most recognizable way. However, there seems to be an eagerness to throw everyone who doesn't otherwise match in the "total newb" bin, and once that happens, an inability to parse facts presented that should indicate that their initial mental models were wrong. ("You can't do that with thousands of threads."/"its dozens of threads" -- (EDIT: and actually, it can be as few threads as one wants.))
I posit this: There are other people like me who have asked non-typical questions, were treated in this way, then just turned their backs on Clojure. I just happen to have a very strong allergy to counterfactuals and make noise in this situation.
Lots of them seem to be stuck in "quick pattern match" mode, so in the other occasions they were "pattern matching" on my asking about a much simpler scenario than I was actually asking about -- then taking my trying to inform them their assumptions were wrong to conclude something like "he doesn't understand threads."
It might be in the way you're asking.
I say as much in my comment above -- I don't know enough about how to ask the questions in the most recognizable way. However, there seems to be an eagerness to throw everyone who doesn't otherwise match in the "total newb" bin, and once that happens, an inability to parse facts presented that should indicate that their initial mental models were wrong. ("You can't do that with thousands of threads."/"its dozens of threads" -- (EDIT: and actually, it can be as few threads as one wants.))
I posit this: There are other people like me who have asked non-typical questions, were treated in this way, then just turned their backs on Clojure. I just happen to have a very strong allergy to counterfactuals and make noise in this situation.
+1 about the Clojure IRC. It's very hard to deal with. I've had a lot of conversations that went something like this:
Me: What's the preferred Clojure solution for this highly specific performance issue that I've tracked down?
IRC: Did you benchmark your code? Have you tried using functions?
Me: groan
Edit: Perhaps there are some people present who want to defend the Clojure IRC channel... I'm being 100% honest with my experiences, only slightly paraphrasing things for humor value. The above conversation actually happened, in a more verbose form.
Me: What's the preferred Clojure solution for this highly specific performance issue that I've tracked down?
IRC: Did you benchmark your code? Have you tried using functions?
Me: groan
Edit: Perhaps there are some people present who want to defend the Clojure IRC channel... I'm being 100% honest with my experiences, only slightly paraphrasing things for humor value. The above conversation actually happened, in a more verbose form.
Right on. The thing about the kind of language weeniedom that smart enlightened people commit, is that there are always smart, enlightened intentions behind it.
IRC is a haven for this personality type. Consider it a last resort.
What other places can I resort to, before reaching IRC?
> Just what kind of community treats its newcomers by ... telling them to go read Lisp macros?
Macros (and DSLs) are more like their own little languages than they are like simple functions. To understand a macro on a fundamental level, you have to feel the need which suggested its creation. And the simplest way to do that, I think, is to read the macro's definition, to see what the code would look like without the macro there to clean it up.
This means, though, that to the extent that a library is definted in terms of macros, you have to meditate upon that library to understand not just how it was written, but why. This is the barrier macros create: without them, you can wave around other people's APIs and code in a cargo-cult fashion, and you'll probably get what you want. Macros prohibit cargo-culting a solution together: you actually have to understand the goals the library was built to serve.
Macros (and DSLs) are more like their own little languages than they are like simple functions. To understand a macro on a fundamental level, you have to feel the need which suggested its creation. And the simplest way to do that, I think, is to read the macro's definition, to see what the code would look like without the macro there to clean it up.
This means, though, that to the extent that a library is definted in terms of macros, you have to meditate upon that library to understand not just how it was written, but why. This is the barrier macros create: without them, you can wave around other people's APIs and code in a cargo-cult fashion, and you'll probably get what you want. Macros prohibit cargo-culting a solution together: you actually have to understand the goals the library was built to serve.
This is exactly the response that is a bad response.
All the OP needed is an explanation of the syntax of a Compojure macro. If someone is asking about the syntax of a specific macro, then "you don't understand macros" is not the correct response.
And that sort of thing doesn't work anyway for 90% of people.
All the OP needed is an explanation of the syntax of a Compojure macro. If someone is asking about the syntax of a specific macro, then "you don't understand macros" is not the correct response.
And that sort of thing doesn't work anyway for 90% of people.
That's what I was saying--macros don't work for 90% of people, because 90% of people aren't willing to read the libraries they use. (I'm not a Clojure user, if you're curious.)
The last time I checked, Compojure didn't have a formal description of its core macros. This might have changed.
Anyway, macros are easy to understand. They're hard to grok, but easy to understand. Generally you grok things by using them.
Anyway, macros are easy to understand. They're hard to grok, but easy to understand. Generally you grok things by using them.
Many things said in that video are kinda off.
He mixes password hashes and MAC (message authentication codes), saying we should HMAC our password hashes. There's little to gain from that:
- Password hashes exist to hide passwords. The attack is to discover the original password by brute force, dictionary, rainbow tables etc. The protection against this is irreducibly slow hash algorithms and unique public salt per hash.
- MAC exists to authenticate message origin. Unlike passwords, the message is often public. The attack is therefore not to discover the message, but to manipulate the message, yet make it appear from the same origin as the original message. The protection against this is the HMAC algorithm with a secret key that only the original message author has.
HMAC-ing your password hash won't stop, or even significantly slow down any of the attacks performed against password hashes, so it feels cargo cultish to claim it's just magically more secure by using it. It's even more WTF-y to claim a password hash library should offer HMAC, or it's insecure.
Once a system is compromised, and the attacker doesn't care to recover the original password, it's far more trivial to just replace the hash or just directly read the data protected by the login directly in the DB, rather than play out a MAC attack on a pass hash. It makes no sense.
HTML escaping. You either escape for HTML or you don't. There's no sane way to escape some of it. His idea that we should be able to escape some tags, but skip (or "strip") others is a huge vector of attack as HTML is notoriously filled with edge cases that your selective escaping/stripping engine won't have. It's not trivial to parse what's a tag and what isn't a tag to a browser, trust me. It only looks trivial if you're ignorant.
If you want to allow some tags, and disable others (say, allow bold/italic on forums, but nothing else) you should go through a strict DSL allowing only those types of formatting, which are then converted to HTML. Just like HackerNews does it - I type star-word-star and I get "word" in italic. That's a DSL. That DSL could be made to look like HTML, but it won't be HTML. It's fully parsed and rebuilt from the DSL syntax tree. So it can be made safe. Selective HTML escaping without those crucial steps is almost impossible to make safe by comparison.
His rant about "there are too many templating engines - we should stop" - yeah, good luck saying we should stop and this working out.
Likewise about databases. "Let's just stop". B.S. The thing he's missing is that different DB engines have different strengths and weaknesses. That's one major reason there are so many, and why many get used in the same project. PgSQL isn't SQLite isn't MongoDB isn't Redis.
By enforcing one type of database for multiple separate project components, you don't improve security, you just cripple architecture and performance.
He mixes password hashes and MAC (message authentication codes), saying we should HMAC our password hashes. There's little to gain from that:
- Password hashes exist to hide passwords. The attack is to discover the original password by brute force, dictionary, rainbow tables etc. The protection against this is irreducibly slow hash algorithms and unique public salt per hash.
- MAC exists to authenticate message origin. Unlike passwords, the message is often public. The attack is therefore not to discover the message, but to manipulate the message, yet make it appear from the same origin as the original message. The protection against this is the HMAC algorithm with a secret key that only the original message author has.
HMAC-ing your password hash won't stop, or even significantly slow down any of the attacks performed against password hashes, so it feels cargo cultish to claim it's just magically more secure by using it. It's even more WTF-y to claim a password hash library should offer HMAC, or it's insecure.
Once a system is compromised, and the attacker doesn't care to recover the original password, it's far more trivial to just replace the hash or just directly read the data protected by the login directly in the DB, rather than play out a MAC attack on a pass hash. It makes no sense.
HTML escaping. You either escape for HTML or you don't. There's no sane way to escape some of it. His idea that we should be able to escape some tags, but skip (or "strip") others is a huge vector of attack as HTML is notoriously filled with edge cases that your selective escaping/stripping engine won't have. It's not trivial to parse what's a tag and what isn't a tag to a browser, trust me. It only looks trivial if you're ignorant.
If you want to allow some tags, and disable others (say, allow bold/italic on forums, but nothing else) you should go through a strict DSL allowing only those types of formatting, which are then converted to HTML. Just like HackerNews does it - I type star-word-star and I get "word" in italic. That's a DSL. That DSL could be made to look like HTML, but it won't be HTML. It's fully parsed and rebuilt from the DSL syntax tree. So it can be made safe. Selective HTML escaping without those crucial steps is almost impossible to make safe by comparison.
His rant about "there are too many templating engines - we should stop" - yeah, good luck saying we should stop and this working out.
Likewise about databases. "Let's just stop". B.S. The thing he's missing is that different DB engines have different strengths and weaknesses. That's one major reason there are so many, and why many get used in the same project. PgSQL isn't SQLite isn't MongoDB isn't Redis.
By enforcing one type of database for multiple separate project components, you don't improve security, you just cripple architecture and performance.
I'm pretty sure the intention behind something like this:
You could replace HMAC with encrypt with AES (or scrypt for that matter) to get the same practical effect.
Edit: clarification
(def a-password (-> pass
(hmac server-stored-key)
(scrypt-kdf work-factor))
You could replace HMAC with encrypt with AES (or scrypt for that matter) to get the same practical effect.
Edit: clarification
I think this is actually works. Might want a configurable work factor for the kdf.
(ns passwords
(:require [pandect.core :refer [sha256-hmac]]
[crypto.password.scrypt :as scrypt]
[environ.core :refer [env]]))
(def SECRET-KEY (if-let [key (env :secret-key)]
key
(throw "Set your SECRET_KEY!!!")))
(defn encrypt-password
[pass]
(-> pass (sha256-hmac SECRET-KEY) scrypt/encrypt))
The problem of lookup table attacks is solved by the public salt I already mentioned. HMAC protects against attacks like length extension (google it), which aren't applicable for discovering a password from a hash.
Encryption also does nothing here, because it's reversible.
Your encryption key is somewhere on your server, so whatever you encrypted (either the plain text password, or the hash) can be decrypted once, and then if it's a hash you just attack the bare hash as usual (brute force, dictionary etc.).
Instead of encryption you can just store part of the salt outside the DB, but with a good hash this is usually not needed and just pointless obfuscation.
Encryption also does nothing here, because it's reversible.
Your encryption key is somewhere on your server, so whatever you encrypted (either the plain text password, or the hash) can be decrypted once, and then if it's a hash you just attack the bare hash as usual (brute force, dictionary etc.).
Instead of encryption you can just store part of the salt outside the DB, but with a good hash this is usually not needed and just pointless obfuscation.
Consider this scenario (which is extremely common):
Public access to a SQL database. Input isn't properly sanitized while building some complicated query. $badnick executes a SQL injection. Luckily, the database user from the public site only has SELECT privileges (yay SOA or something). This is an extremely common enterprise situation.
The attacker can dump data (like password hashes), but in an HMAC'd password setup, cannot actually gain access to modify or login as a user via brute-forcing. They cannot pivot the user account to access to another system. The attacker must find a different hole to gain access to the app/login server.
Public access to a SQL database. Input isn't properly sanitized while building some complicated query. $badnick executes a SQL injection. Luckily, the database user from the public site only has SELECT privileges (yay SOA or something). This is an extremely common enterprise situation.
The attacker can dump data (like password hashes), but in an HMAC'd password setup, cannot actually gain access to modify or login as a user via brute-forcing. They cannot pivot the user account to access to another system. The attacker must find a different hole to gain access to the app/login server.
Well first of all your "common scenario" terrifies me, and betrays an app developed by incompetent devs who wouldn't even know what a "hash" is, because in any sane situation:
1) The SQL database is not public (any more than it's common to have public anonymous FTP to your server's app deployment). It listens on a specific interface, specific port, from specific IPs with a specific user and password (which is not "root"/"").
2) Input isn't properly sanitized? Hello? If that's "common" then fix that first. It's not 1998 anymore, no one has an excuse for that level of ignorance. First of all "sanitize" is an incorrect term for what needs to be done. You either escape an SQL literal, or you pass it as a parameter to a prepared statement. You do this to all data, all DB layers provide escaping, and it's absolutely trivial to do. And for prepared statements, there is no way to do it wrong, because it's not you who does it. Data and query in prepared statements are separated at the protocol itself and injection is impossible, you can't possibly mess it up. All of this is "using databases 101". Heck, your app will break even with innocent data containing, say, a quote, if you fail those.
3) I mentioned twice already, there's public salt for hashing the password, and there's irreducibly slow hash to be used to protect against a weak password (like Blowfish). This is what breaks brute-force attacks, not HMAC.
4) Let's assume your exact scenario is indeed terrifyingly common. Still, HMAC does nothing at all here, that keeping part of the salt off-the-DB won't do. HMAC doesn't mean "when you have a key outside the DB". You don't need HMAC to keep part of the salt outside the DB. Just... keep part of the salt outside the DB, without HMAC.
But if you feel the need to obfuscate your code like this, it means you're working in hell and some people don't deserve their jobs, and better fix that, before you start thinking about compensating for it at the wrong place of the app.
1) The SQL database is not public (any more than it's common to have public anonymous FTP to your server's app deployment). It listens on a specific interface, specific port, from specific IPs with a specific user and password (which is not "root"/"").
2) Input isn't properly sanitized? Hello? If that's "common" then fix that first. It's not 1998 anymore, no one has an excuse for that level of ignorance. First of all "sanitize" is an incorrect term for what needs to be done. You either escape an SQL literal, or you pass it as a parameter to a prepared statement. You do this to all data, all DB layers provide escaping, and it's absolutely trivial to do. And for prepared statements, there is no way to do it wrong, because it's not you who does it. Data and query in prepared statements are separated at the protocol itself and injection is impossible, you can't possibly mess it up. All of this is "using databases 101". Heck, your app will break even with innocent data containing, say, a quote, if you fail those.
3) I mentioned twice already, there's public salt for hashing the password, and there's irreducibly slow hash to be used to protect against a weak password (like Blowfish). This is what breaks brute-force attacks, not HMAC.
4) Let's assume your exact scenario is indeed terrifyingly common. Still, HMAC does nothing at all here, that keeping part of the salt off-the-DB won't do. HMAC doesn't mean "when you have a key outside the DB". You don't need HMAC to keep part of the salt outside the DB. Just... keep part of the salt outside the DB, without HMAC.
But if you feel the need to obfuscate your code like this, it means you're working in hell and some people don't deserve their jobs, and better fix that, before you start thinking about compensating for it at the wrong place of the app.
I should clarify that when I say "public access to", I mean specifically that a public person can access an application talking to the SQL database.
I.e. User <-> App <-> Database
Input is simply not properly sanitized in the real world because mistakes happen. This isn't by design. This is a source of remote execution. The remote execution only yields read access since the app only needs read access (and thus the fewest privileges possible are read access). The attacker can read password hashes, but cannot update them.
As I mentioned before, HMAC is just an example of keeping all information necessary to login to the site out of the stored hash. Any non-constant value per password will do (feel free to maintain a lookup table of unique lists to active passwords in your app if you feel you must, I'll be HMAC'ing the user's input password against a constant secret key before sending to KDF). The idea is that an attacker who can read your database (but not the application code) cannot gain access as a user without first also compromising the application code.
I.e. User <-> App <-> Database
Input is simply not properly sanitized in the real world because mistakes happen. This isn't by design. This is a source of remote execution. The remote execution only yields read access since the app only needs read access (and thus the fewest privileges possible are read access). The attacker can read password hashes, but cannot update them.
As I mentioned before, HMAC is just an example of keeping all information necessary to login to the site out of the stored hash. Any non-constant value per password will do (feel free to maintain a lookup table of unique lists to active passwords in your app if you feel you must, I'll be HMAC'ing the user's input password against a constant secret key before sending to KDF). The idea is that an attacker who can read your database (but not the application code) cannot gain access as a user without first also compromising the application code.
Once again there's no way to make a mistake with a prepared statement. The only way to do it is using poor practices like using half baked "sanitizing" functions which no competent developer will use.
And once again, you don't need HMAC to store salt outside the hash. You just don't. HMAC doesn't dictate where you store your salt. You're bundling these two things together as if they're inseparable, but they're two completely separate things.
And once again, you don't need HMAC to store salt outside the hash. You just don't. HMAC doesn't dictate where you store your salt. You're bundling these two things together as if they're inseparable, but they're two completely separate things.
Security's something you do in-depth. You might build a perfect app, which correctly sanitizes your DB inputs (and you're right, it's easy these days). That won't necessarily stop $developer making a little PHP interface long after you leave, which exposes your DB again.
I'm not up to the play with HMAC so can't confidently comment on that, but the point being made is that there is lots of SQL injection out there. Keeping the salt out of the DB seems like a not-insane way to help mitigate a compromise.
I'm not up to the play with HMAC so can't confidently comment on that, but the point being made is that there is lots of SQL injection out there. Keeping the salt out of the DB seems like a not-insane way to help mitigate a compromise.
> "That won't necessarily stop $developer making a little PHP interface long after you leave, which exposes your DB again."
Let me ask, does this hypothetical company have someone in charge of architecture, or everyone just codes randomly entirely on their own with no rhyme or reason until it works? Because that's what you're describing. Anarchy & chaos, with no one in charge.
The answer to the above hypothetical situation is that the PHP developer will code to the service layer, not to the database. In fact, that precise setup is very common in the projects I do: services (say in Java) consumed by delivery mechanisms (say PHP web sites).
Multiple independent apps reaching into the database directly is not only recipe for a security disaster, but a recipe for data integrity and maintenance disaster.
Here I explained service layers, so I won't repeat myself: https://news.ycombinator.com/item?id=7476725
Let me ask, does this hypothetical company have someone in charge of architecture, or everyone just codes randomly entirely on their own with no rhyme or reason until it works? Because that's what you're describing. Anarchy & chaos, with no one in charge.
The answer to the above hypothetical situation is that the PHP developer will code to the service layer, not to the database. In fact, that precise setup is very common in the projects I do: services (say in Java) consumed by delivery mechanisms (say PHP web sites).
Multiple independent apps reaching into the database directly is not only recipe for a security disaster, but a recipe for data integrity and maintenance disaster.
Here I explained service layers, so I won't repeat myself: https://news.ycombinator.com/item?id=7476725
in any sane situation:
1) The SQL database is not public (any more than it's common to have public anonymous FTP to your source code). It listens to a specific interface, from specific IPs with a specific user and password (which is not "root"/"").
Yes, mostly true, but in the common case passwords are shared between users, and in corporate apps there are some widely shared "admin" accounts that are user accounts with some added capabilities, or some such bad practice.
you pass it as a parameter to a prepared statement. You do this to all data, all DB layers provide escaping, and it's absolutely trivial to do. And for prepared statements, there is no way to do it wrong
Yeah, so what exactly is wrong with SQL Korma? I thought that would be one succinct and abstracted way of making sure all requests are through prepared statements.
1) The SQL database is not public (any more than it's common to have public anonymous FTP to your source code). It listens to a specific interface, from specific IPs with a specific user and password (which is not "root"/"").
Yes, mostly true, but in the common case passwords are shared between users, and in corporate apps there are some widely shared "admin" accounts that are user accounts with some added capabilities, or some such bad practice.
you pass it as a parameter to a prepared statement. You do this to all data, all DB layers provide escaping, and it's absolutely trivial to do. And for prepared statements, there is no way to do it wrong
Yeah, so what exactly is wrong with SQL Korma? I thought that would be one succinct and abstracted way of making sure all requests are through prepared statements.
Thanks for this clarification on HMAC - I was also failing to see and understand the relevance to passwords when investigating after watching the talk.
I support a Clojure/ClojureScript web app in my day-to-day work. I've also written an OAuth2 workflow for friend (https://github.com/ddellacosta/friend-oauth2 ...pull requests welcome), one library Aaron Bedra spends a lot of time talking about. The story really is exactly how he describes--Clojure web security could be better. I suppose you could argue that, at least, the defaults are generally not bad...but that's because there aren't defaults in many cases.
I think the main issue is, ironically, the same thing that makes Clojure development so great: you end up composing a bunch of small libraries together, each of which do one thing well. But as Aaron points out, there are things that aren't taken care of and you can't assume they are done for you, and a lot of security is something that you don't technically need to get a web app working...
I wouldn't want to give up the approach of composing a bunch of libraries to build concise, easy-to-reason-about web apps in Clojure--I think it helps with security in and of itself--but the community could really benefit from having more libraries which nail these things down and help developers build web apps more safely, with less work.
That said, I don't think it's all bad: you can build high quality, secure web apps in Clojure, it just requires you to really pay attention to security. And we should be doing that anyways, right?