If the server cannot be trusted, it will extract your encrypted data, since it serves up the code. The server, if compromised/subpoenaed, merely needs to serve you some JavaScript that sends home the encryption key, and your data is no longer just your data.
We are still working on new sets, though obviously the rate of new sets is pretty low. The mailing list is basically unmonitored at this point, but everything we've got is on the site. (This is a vast improvement on the previous state, where we regularly failed to set out challenges to people who emailed us, due to overload.)
> Penetration tests, when done by a good firm like Matasano, are incredibly useful, but lose their value the next time you push code.
I'd like to nicely but firmly push back on this one, and have longitudinal analysis of clients' applications to back it up. We put a lot of effort into helping our customers improve over time, both formally (writing helpful recommendations) and informally (educating developers during and after the test). There exist customers that ignore our advice, and don't improve, but most have a dramatic improvement in new code quality after the first assessment, and continue to year after year.
Note that our work-sample tests are, not-insanely, done in the comfort of your own home, at your own pace, on your own schedule, and represent the work we actually do. As co-head of recruiting for NCC US (aka head of recruiting for Matasano), I think the answer to the in-person interview question of "write code to do this," is "thank you for your time, I'll see myself out."
Most of our candidates drop out before the work sample. On the other hand, almost none of our candidates are qualified to work for us when they initially apply.
We make it really clear that there will be work samples before people even apply. I think the real question would be "how many qualified applicants don't bother applying because of the work sample," which is a question we can't answer. Given the paucity of unemployed qualified infosec folks, we're comfortable with the tradeoff.
Here are a couple of resources that I tend to hand out to startups that we do work for at Matasano. No charge :-)
Not trying to be a salesperson, but I feel like most startups get more value out of sitting down with a security consultant for a couple days and talking about architecture and dev processes then they do getting a full penetration test. Like the presentations say, the big risk in the early days is lack of interest, not security. I feel like a startup's big security concern it doing something that's going to make them have to rewrite everything later on.
In general, my feeling is that the Matasano process (which I currently manage) works outstandingly well where there isn't a flood of qualified candidates. If you have a glut of folks who are ready to start working, you can get away with a terrible process.
We do free-form technical interviews, but only to try to detect candidates who really aren't ready for the work-sample challenges. Our in-person interviews are standardized and try to evaluate consulting/architecture skills that are hard (impossible?) to measure without people. These involve open-ended intermediary questions, but the final answers are structured.
The bottom line is this: you cannot compare candidates using free-form interviews. You must compare candidates who are going to be doing similar work. Thus, free-form interviews have no evaluative value.
My policy (I'm co-in-charge of recruiting at Matasano/NCC, and a lot of folks report to me) is this:
1) Hire based on current ability, not potential. Hiring based on potential is a minefield that our whole process is designed to avoid. ("Enthusiasm," "cultural fit," talking a good game without being able to walk the walk, etc.)
2) Be aggressively fair in giving out promotions and raises. Keeping someone's salary low because they're bad at asking for raises is not a good long-term strategy, especially in consulting.
We actually pre-pay on that. Candidates get an initial call with a very senior person to start. That call includes coaching on how to get through our interview process, and concludes with us sending free educational material (books, etc.). This coaching and material is not specific to getting a job at Matasano, you can just as easily use it to get a job at one of our competitors. We invest hundreds of dollars in each candidate before we ask them to do our challenges.
Getting an internship at Matasano is HARD. Unlike normal hiring, we are limited in the number of spots we can offer, and we also have a huge flood of candidates at once. I hate that it's so hard, and that we can't provide our usual standard of support to interview applicants. There's just no way for us to scale with the seasonal demands for summer interns.
Please please please apply for full-time when you graduate! The process is a lot smoother.
The way we do it now, the initial call person reads your resume, but we categorically do not reject based on the initial call (or the subsequent tech phone interview(s)). So, we get the advantage of being able to talk to someone about their experience and helping them make a plan to get hired, but also the advantage that our process is actually resume-blind.
At Matasano (slash NCC), we get a lot of candidates. We look at resumes so that we have something to break the ice with when we talk to the candidates. We do triage interns using resumes, because we get hundreds over the course of a few weeks, but never for full-time candidates.
How do we figure out which ones are serious? We talk to them, usually give them homework, and then give them work-sample tests. This is not particularly expensive for us, and gives us repeatable, relevant, apples-to-apples metrics on how qualified they are to do the work. This is, as Tom insists, surprisingly easy for us. (Not to say it's easy, but nothing about hiring is easy.)
Hi, I've taken over from Tom for hiring at Matasano. There's a couple of things that you need for "diverse" recruiting (e.g. hiring women in tech):
1) A way of evaluating candidates that avoids mirrortocracy style institutional -isms like sexism and ageism.
2) A way of convincing people who have bounced off the field due to -isms to even try.
BTW, Tom was great at Matasano on #1, and only moderately successful at #2. We get tons of people who aren't in the industry, but they tend to self-select to be young men. For an individual company, #2 is way harder. It's "easy" to fix yourself, but hard to fight the larger culture.
Starfighter looks to me like something that supports #1 directly (by allowing objective metrics), and enables #2. It's a way for people who aren't welcome into a field to dip their toe in without some roomful of young white men asking them illegal questions about their child-rearing plans. On the other hand, there's still tremendous pressure pushing people away, and it's a deep problem. Still, part of the puzzle and I (personally) highly endorse.
As to the numbers, I've been meaning to dig into the data, but Matasano's hiring as fast as we need to, so it's been hard to motivate myself to do so. Maybe once graduation season ends...
These days we mostly send The Web Application's Hacker's Handbook and a link to microcorruption. (We do somehow get candidates which haven't heard of microcorruption.) Generally, we continue to endorse Tom's Amazon reading list: http://www.amazon.com/An-Application-Security-Reading-List/l...
Speaking as one who stands to benefit from such a rule, I also think that requiring 3rd party validation is a bad idea. First off, it's always a race to the bottom, and secondly, there are not enough qualified people in the world to look at everything.
I would, however, like to see a general rule requiring software and hardware makers to take "reasonable steps" to secure their products, and opening them up to liability if they do not. A few class-action lawsuits would go a long way towards encouraging everyone to put in a secure SDLC.
Certainly not required! To get a job in application security (at Matasano/NCC or anywhere, really) you should be demonstrably okay at web application, and have interests beyond webapps.