Hiring Great Developers: The Pre-Screening(small-improvements.com)
small-improvements.com
Hiring Great Developers: The Pre-Screening
http://www.small-improvements.com/recruiting/hiring-smart-developers--pre-screening
14 comments
Wrote it in the post comment, but will write again. The company I helped get traction (Codility) would address many issues people have with remote testing: manual work, some of the cheating and qualitative vs quantitative dilema. I may be biased, due to my former involvement, but it costs nothing to try it, so you can see for yourself: http://codility.com/demo/take-sample-test/
Bunging in my own http://codeboff.in in the same spirit.
Upvoted you both for on-topic self-promotion. HN needs more of that.
An interesting side note to gather from this is that it confirms something I've long suspected: LinkedIn recommendations are useless. FWIW I have a couple good ones, but I don't think they've ever even helped me get an interview much less a job. I don't think anybody who actually makes hiring decisions has time to look at them; seems the only purpose they serve is to let people who casually cruise your profile know that you are competent.
Posts like these reenforce my dismay at the total and utter devaluation of the education system when dealing with recruiters for full-time positions (15 years minimum of full-time education, but you still need that one final test).
That said, I don't have a problem with these tests if the setter understands that they are not there to filter in or out the competent, but instead serve as a filter of convenience because dealing with 60 resumes is easier than 600. This seems an acceptable pragmatic approach.
When rubber hits the road a business needs to make money by being competitive. If having read a book on JUnit is your metric for hiring competitive employees, so be it. I can't help but wonder if there is a sizeable amount of latent competitive advantage in the complement of this set.
That said, I don't have a problem with these tests if the setter understands that they are not there to filter in or out the competent, but instead serve as a filter of convenience because dealing with 60 resumes is easier than 600. This seems an acceptable pragmatic approach.
When rubber hits the road a business needs to make money by being competitive. If having read a book on JUnit is your metric for hiring competitive employees, so be it. I can't help but wonder if there is a sizeable amount of latent competitive advantage in the complement of this set.
it isn't about process, it is about people. As Steve Jobs put it : "A players hire A players, B players hire C players."
http://www.folklore.org/StoryView.py?project=Macintosh&s...
http://www.folklore.org/StoryView.py?project=Macintosh&s...
Interspersing all kinds of rebuttals to complaints others have made about the post inside the post itself looks really defensive. It disrupted my reading.
> I am asking applicants to write some unit tests, to summarise the key point of a Google I/O video presentation that's actually fun to watch. I am asking them to learn enough about a web framework to be able to write a very basic program. I am asking them to review some really dumb code. And since I don't need code-monkeys but people with opinions, I am asking them for their opinion about my protoype: What would you improve?
Shit you are kidding right? Or do people just really value their time that low?
Shit you are kidding right? Or do people just really value their time that low?
A fine article for hiring adequate or average developers.
You can't really hire great developers using any process involving resumes and ads. Hiring great developers involves recruitment. You already know you would hire the developer before you even contact them. The entire issue is can you offer them enough to lure them from where they are now.
You can't really hire great developers using any process involving resumes and ads. Hiring great developers involves recruitment. You already know you would hire the developer before you even contact them. The entire issue is can you offer them enough to lure them from where they are now.
Or, it should be seen more as a tool for filtering people out rather than for selecting people in. And if they are used, these tests should be (1) done at the very earliest stages in the process and (2) be conducted in a manner that's as time-efficient as humanely possible.
And just to be sure, we disregarded the results from the screening anyway, and based our hiring decisions only on the three onsite-interviews.
Am I missing something here?
Am I missing something here?
I suspect it means passing the screening gets you through to the onsite-interviews, but the subsequent decision about hiring is purely based on the interviews, not on any results or rankings from the screenings.
The screening process he's describing is just to throw applications away, not to actually accept anyone.
We tried it and while it does help to filter out completely clueless people, however cheating is a real problem. This clearly depends on the industry and specifics of the position, but out of three people that hired, two clearly cheated on the remote coding test. This was for a C position in embedded context, and one would think that no one in their sane mind would cheat, and still many people did.
So, yeah, good idea, but use with caution as YMMV.
So, yeah, good idea, but use with caution as YMMV.
We tried it and cheating happens, but for us it was bearable. We were running candidates through Codility as the very first level of screening (3 programming problems to be solved from home), then through 2 or 3 interviews. People were cheating on a test once in a while, but not badly enough to spoil the whole filtering process. At some point we started sending out a welcome e-mail before the screening session to make it clear that cheating is nothing more than a recipe for a seriously bad interview, that you really do not want to have. Helped somewhat, but still somebody cheats once in a while. I guess such is life.
> out of three people that hired, two clearly cheated on the remote coding test
What screening did you do onsite?
What screening did you do onsite?
No, all from other cities. Sent the task over email and wanted to get a solution back in 8 hours or less.
Right, I mean after the remote coding test, did you bring them onsite for another battery of screening?
We had several phone interviews - one technical before the test, one - after and another one with HR, and then CTO insisted on hiring and flying them over right away. With a three month probation period... that did come handy. So basically the first two weeks were the on-site screening if you will, and which they spectacularly failed.
Ok, that makes a lot more sense then.
Here's a much better approach: use IRC.
I've successfully recruited dozens of exceptional individuals through IRC and am surprised that it's not often used for recruitment. Networks like freenode are filled with amazing individuals who are quite amenable to interesting startups. Even with my first company back in '99, when I didn't even know how to code, I found that the developers hired through IRC performed magnitudes better than the ones who came through more traditional hiring processes.
Just hang out for a few months in the IRC channels of open source projects that you find relevant; get to know the various individuals; once you find someone that you believe you can work well with, engage them about joining your venture. Easy, effective and the social bond even lasts for years afterwards.
I've successfully recruited dozens of exceptional individuals through IRC and am surprised that it's not often used for recruitment. Networks like freenode are filled with amazing individuals who are quite amenable to interesting startups. Even with my first company back in '99, when I didn't even know how to code, I found that the developers hired through IRC performed magnitudes better than the ones who came through more traditional hiring processes.
Just hang out for a few months in the IRC channels of open source projects that you find relevant; get to know the various individuals; once you find someone that you believe you can work well with, engage them about joining your venture. Easy, effective and the social bond even lasts for years afterwards.
But I am still surprised at how few companies do this
Because it flags you as a mill, uninterested in the applicant and likely a firm that treats development as a commodity. I'd be willing to bet that the 30% who didn't bother to respond to the challenges included some of your best applicants.
You'd never hire an accountant by insisting they prepare a quarterly for you before you ever met them and you wouldn't make prospective truck drivers arbitrarily drive hundreds of miles to an interview.
If you want to screen for the absolute basics of rote capability that's what those various certified professional type credentials are for. But just like your little tests, you'll be pre-screening out the best as well as the worst.
Because it flags you as a mill, uninterested in the applicant and likely a firm that treats development as a commodity. I'd be willing to bet that the 30% who didn't bother to respond to the challenges included some of your best applicants.
You'd never hire an accountant by insisting they prepare a quarterly for you before you ever met them and you wouldn't make prospective truck drivers arbitrarily drive hundreds of miles to an interview.
If you want to screen for the absolute basics of rote capability that's what those various certified professional type credentials are for. But just like your little tests, you'll be pre-screening out the best as well as the worst.
"Because it flags you as a mill, uninterested in the applicant and likely a firm that treats development as a commodity."
My impression is exactly the opposite. It flags them as a company that cares about technical competence.
"If you want to screen for the absolute basics of rote capability that's what those various certified professional type credentials are for."
Their test screen for more than that. And of course those professional certifications are useless if you go beyond the very basic level, just like "CV-years of experience" are useless for determining competence.
My impression is exactly the opposite. It flags them as a company that cares about technical competence.
"If you want to screen for the absolute basics of rote capability that's what those various certified professional type credentials are for."
Their test screen for more than that. And of course those professional certifications are useless if you go beyond the very basic level, just like "CV-years of experience" are useless for determining competence.
Most of the most talented won't have those credentials and, in my opinion unlike getting sample problems, being asked for the credentials is such a red flag that the best won't apply.
I agree with that. The problem with third party certs which you allude to is they are almost exclusively taken by mediocre developers looking for an easy feather in their cap. Meanwhile true talent has no reason to bother with such nonsense.
The credential I look for when hiring is someone who has already done something remarkable.
The credential I look for when hiring is someone who has already done something remarkable.
It flags them as a company that cares about technical competence
I bet the tasks being sent out are most probably something in the vein of parsing a CSV file or (in case of Atlassian) writing a simple JSP or trivial XSLT or what have you. Not the best check for competence.
I bet the tasks being sent out are most probably something in the vein of parsing a CSV file or (in case of Atlassian) writing a simple JSP or trivial XSLT or what have you. Not the best check for competence.
The company I work for has prospective developers write something that parses Java property files, and sample files are given to them to examine. It is amazing how many are unable to write something to do this. Additional small-grained programmer behaviors are also discovered besides if they can code the answer: do they comment their code, do they structure it, name it in a thought out manner or use X,Y, and Z as variable and function names? Do they use exception handling? It is a pretty useful test and filters out people who have head knowledge but inability to do a simple task, or have impressive resumes that mean nothing.
Atlassian isn't hiring people solely on the strength of their screener. They just want to be able to more objectively screen out people who have no business applying. Most companies do this by reviewing resumes and getting rid of everyone who, say, doesn't have X years experience with Y language, or didn't have Z GPA in college.
You should give it a try, see what they send you! The stuff we did when I was still there was definitely more advanced. But again, pre-screening is never a test for competence, it is a test for lack of competence. The real test for great skill comes in the form of a 90m+ coding test and a 2h technical interview.
Tests taken remotely do not work to screen out the incompetent who routinely take them to the internet for solutions or pay someone else to solve. Nor do they screen in the competent who realize their day off work is much better spent making some phone calls or networking if they are really looking to switch jobs rather than spend it doing things like learning a new web framework to solve some canned problem.
And yes, he actually mentions in the article this is an actual thing he does: have applicants learn new frameworks in order to apply. Sheesh. Who has time for that except the very desperate.
And yes, he actually mentions in the article this is an actual thing he does: have applicants learn new frameworks in order to apply. Sheesh. Who has time for that except the very desperate.
I tried to address most concerns on my post. But I missed your comment about the accountant and the truck driver. I'd never ask a truck driver to travel hundreds of miles.
In fact, bringing in people to an interview right away without at least calling them IS like asking the truckie to drive a long distance just for the sake of it. I'd avoid that. But I would ask him to send me a photo of his current truck (if he claims he has one). I'd maybe also ask him to name five roads he'd avoid between his main destinations (say in SF, if he claims to be from SF). Someone who's actually a cab driver in NY would find that hard to answer. :-)
In fact, bringing in people to an interview right away without at least calling them IS like asking the truckie to drive a long distance just for the sake of it. I'd avoid that. But I would ask him to send me a photo of his current truck (if he claims he has one). I'd maybe also ask him to name five roads he'd avoid between his main destinations (say in SF, if he claims to be from SF). Someone who's actually a cab driver in NY would find that hard to answer. :-)
There's a major issue with your method. The best developers know they can get jobs wherever they want. What incentive do they have to bother with the extra work?
Because, a) the 'best developers tend to love beating challenges so see it as a fun exercise to do one evening, and b) good developers want to work with good developers and seeing the screening process is a benefit.
Good developers of a certain experience level generally realize that if they apply for a job somewhere and waltz through the interview, or worse, the interview is a basically a pleasant chat with the tech director, then the chances are the hiring standards really aren't that great .
Oh and c) - if a developer does feel they're too good for your screening test, or such a thing is enough to put them off your company, then weeding them out at such an early stage may actually be a positive thing.
I do think it's crass to send out a problem sheet without at least verbally talking to the candidate briefly though. We currently do -
5-15 minute chat with a recruiter or PM who is savvy enough to filter out the people who really aren't experienced enough.
3 hour timed problem (debugging, coding, math, domain knowledge)
Technical phone screen
In-person interview (6-7 hours, all expenses paid).
Good developers of a certain experience level generally realize that if they apply for a job somewhere and waltz through the interview, or worse, the interview is a basically a pleasant chat with the tech director, then the chances are the hiring standards really aren't that great .
Oh and c) - if a developer does feel they're too good for your screening test, or such a thing is enough to put them off your company, then weeding them out at such an early stage may actually be a positive thing.
I do think it's crass to send out a problem sheet without at least verbally talking to the candidate briefly though. We currently do -
5-15 minute chat with a recruiter or PM who is savvy enough to filter out the people who really aren't experienced enough.
3 hour timed problem (debugging, coding, math, domain knowledge)
Technical phone screen
In-person interview (6-7 hours, all expenses paid).
What you say is true for mediocre to average developers.
Before I had my own company, every place I worked whenever I would talk to clients or go to conferences I would get on the spot job offers. Principals would ask "Would you ever consider leaving XYZ?" I would say I'm not looking at the moment. They would always then hand me their cards and say "If you're ever looking to move, give me a call first."
In addition, I would get calls at home to my unlisted number because of articles I had written or patents I held.
When it was time to move on I would have dozens of open offers already to chose from. Why the hell would I or any other great developer bother dredging through monster hoping to find the 0.001% of ads that are both legitimate and for a place than isn't some web shop with slave hours and low wages? Answer is I wouldn't. Any company that only relies on job ads and resumes is not a place that great developers work for.
Before I had my own company, every place I worked whenever I would talk to clients or go to conferences I would get on the spot job offers. Principals would ask "Would you ever consider leaving XYZ?" I would say I'm not looking at the moment. They would always then hand me their cards and say "If you're ever looking to move, give me a call first."
In addition, I would get calls at home to my unlisted number because of articles I had written or patents I held.
When it was time to move on I would have dozens of open offers already to chose from. Why the hell would I or any other great developer bother dredging through monster hoping to find the 0.001% of ads that are both legitimate and for a place than isn't some web shop with slave hours and low wages? Answer is I wouldn't. Any company that only relies on job ads and resumes is not a place that great developers work for.
Sure, but keep in mind people have busy schedules.
Recently a startup I was somewhat interested in required a 4-5 hour coding project that was similar to the labs I do in school. I'm already bogged down in work interviewing, doing normal classes, writing a senior thesis, and preparing my research for publication.
I decided to forget about the company and look elsewhere, for I already had offers from great places like Google and Facebook and could always fall back.
I'm not saying its bad to screen candidates with interesting puzzles. I'm saying that you risk filtering out the better candidates that just don't have time to deal with puzzles from a company they don't know.
Additionally, keep in mind that I come from a competitive university in the academic world (although I figure competition is pretty bad at most universities). Thus, with my current 100 hour work weeks, I would just rather not bother with solving tedious puzzles.
Recently a startup I was somewhat interested in required a 4-5 hour coding project that was similar to the labs I do in school. I'm already bogged down in work interviewing, doing normal classes, writing a senior thesis, and preparing my research for publication.
I decided to forget about the company and look elsewhere, for I already had offers from great places like Google and Facebook and could always fall back.
I'm not saying its bad to screen candidates with interesting puzzles. I'm saying that you risk filtering out the better candidates that just don't have time to deal with puzzles from a company they don't know.
Additionally, keep in mind that I come from a competitive university in the academic world (although I figure competition is pretty bad at most universities). Thus, with my current 100 hour work weeks, I would just rather not bother with solving tedious puzzles.
When I see application processes like this I'm generally more interested because it signals to me that the company cares about hiring and that I'm more likely to have colleagues there that have been selected for ability rather than having exactly the right checkboxes on their CVs.
I second this. I just graduated and as I was looking for a job and I interviewed for a bunch of companies.
One of them asked that I solve some programming challenges. Sure, I like puzzles and I like being challenged. However, at the time I was already being challenged by my thesis work, problem sets, interviews... it's hard to agree to make a huge time commitment just to get the right to a first round interview.
Since other hot startups/ cool companies (Google, FB) don't require this type of activity, it makes sense to me to think that a lot of people simply won't have the time to apply because they're busy solving more interesting puzzles.
One of them asked that I solve some programming challenges. Sure, I like puzzles and I like being challenged. However, at the time I was already being challenged by my thesis work, problem sets, interviews... it's hard to agree to make a huge time commitment just to get the right to a first round interview.
Since other hot startups/ cool companies (Google, FB) don't require this type of activity, it makes sense to me to think that a lot of people simply won't have the time to apply because they're busy solving more interesting puzzles.
Facebook are well known for their application puzzles,
http://www.facebook.com/careers/puzzles.php
And Google are notorious for having a long drawn out interviewing process.
http://www.facebook.com/careers/puzzles.php
And Google are notorious for having a long drawn out interviewing process.
Facebook's interview puzzles aren't a prerequisite requirement to getting a job. The good developers who don't have time will just interview and get offers.
Google's are drawn out, but each interview takes 1 hour max so there is an upper bound on your time.
Google's are drawn out, but each interview takes 1 hour max so there is an upper bound on your time.
I'd be glad to do the extra work (well yeah, within some limits, I won't code you a website for free :) ) The benefit to me as a developer is that this reduces the probability I'll work with incompetent people.
One major argument I'm hearing is that by giving these coding projects, competent developers should be more interested because this guarantees the people they work with will be competent.
My response is that you can find out the competence of an individual without wasting 5 hours of their time on a pre-interview puzzles. If they pass your phone screen and interview, you should be fairly confident that they are a competent programmer. I'm hard pressed to believe that someone who passes famously difficult interviews (for example Palantir) would find the puzzle anything more than busy work.
My response is that you can find out the competence of an individual without wasting 5 hours of their time on a pre-interview puzzles. If they pass your phone screen and interview, you should be fairly confident that they are a competent programmer. I'm hard pressed to believe that someone who passes famously difficult interviews (for example Palantir) would find the puzzle anything more than busy work.
At my previous job (which I enjoyed greatly until the president embezzled funds and ran off with a woman), I asked to do a programming task so that I could prove my skill and ability to do what they needed.
I simply give them a call and ask them what the most interesting papers they have read over the last year are. Without any forewarning of course. People who are passionate about things know their stuff; and it's not from blog posts or forums that they obtain this information.
And I can tell you, if I learn something while on the phone with them then it's a guaranteed good hire.
And I can tell you, if I learn something while on the phone with them then it's a guaranteed good hire.
I believe that with your approach you are guaranteed to only invite great applicants, so that's cool. But I worry that you also dismiss many great applicants who are not that snappy, have a bad day (minute, I guess!) I couldn't come up with a great paper. I read blogs, sure, but papers?
If I needed a developer who's great at answering tricky questions in front of an audience, your approach is perfect. But I'd also love a developer who solves hard questions with a bit more time to ponder the problem space. I worry I couldn't spot them in a such a tough phone interview. Oh well, next post! :-)
If I needed a developer who's great at answering tricky questions in front of an audience, your approach is perfect. But I'd also love a developer who solves hard questions with a bit more time to ponder the problem space. I worry I couldn't spot them in a such a tough phone interview. Oh well, next post! :-)
No offence but discussing the most recent/interesting paper you have read shouldn't be a tricky question.
Very cool. I'm thinking it would be great if elance-type of sites, or even experts exchange, would have an area where a prospective employer can submit small coding questions to potential employees.