Ask HN: How do you identify a great software developer in an interview?
Several young (fresh out of college) engineers on my team that I interviewed last year impressed me with their intelligence by way of excellent algorithmic thinking. On my recommendation, they were hired. However, several months into the job, they continue to ask trivial (read: stupid) questions, and don't seem to want to figure things out on their own. Yet other engineers, who I was not impressed by during our interview, have absolutely exceeded my expectations. They have the spirit of a hacker; they dig deep into problems, learn everything within our code base, and are invaluable contributors. How do I identify those people during an interview? Why do my intuitions seem backwards?
17 comments
Despite how hard you try, you can't get this 100% right. This is how I approached hiring:
1. Weed out the truly incompetent. A very simple programming exercise is sufficient. Keep in mind that the goal of the exercise is not to identify a great programmer, but rather to identify those who can't program at all.
2. Set expectations. During the interviewing process, try your best to work out what you would expect from the individual if hired. Would you expect them to churn out a ton of quality code? Do you expect them to be great at determining business needs and writing less code, but code with a high impact? Is this going to be your go to guy for technology X? Do you expect your junior person to learn Y and Z so they can start contributing after N months?
3. Communicate your expectations clearly. Make sure the candidate knows what's expected of him or her. Towards the end of the interview discuss how they feel about those expectations. Ask how they plan on going about meeting them.
4. After hiring: If you get it right great, if not, let the person go sooner than later. You do yourself and the individual a great disservice if you try to turn them into something they are not. Don't try to turn a lump of coal into a diamond. It's a trap. It doesn't work. They will be unhappy and resentful if you try. Nobody likes to be fired (or do the firing) but you have to tell yourself it's best that they get back out there and find the job that they can be happy and successful at.
1. Weed out the truly incompetent. A very simple programming exercise is sufficient. Keep in mind that the goal of the exercise is not to identify a great programmer, but rather to identify those who can't program at all.
2. Set expectations. During the interviewing process, try your best to work out what you would expect from the individual if hired. Would you expect them to churn out a ton of quality code? Do you expect them to be great at determining business needs and writing less code, but code with a high impact? Is this going to be your go to guy for technology X? Do you expect your junior person to learn Y and Z so they can start contributing after N months?
3. Communicate your expectations clearly. Make sure the candidate knows what's expected of him or her. Towards the end of the interview discuss how they feel about those expectations. Ask how they plan on going about meeting them.
4. After hiring: If you get it right great, if not, let the person go sooner than later. You do yourself and the individual a great disservice if you try to turn them into something they are not. Don't try to turn a lump of coal into a diamond. It's a trap. It doesn't work. They will be unhappy and resentful if you try. Nobody likes to be fired (or do the firing) but you have to tell yourself it's best that they get back out there and find the job that they can be happy and successful at.
The answer is obvious and right before your eyes: you can't.
This problem has been examined again and again. The only way to find people with sufficient skill in any field is to test them (formal graded test, not idiosyncratic problems).
No one has ever demonstrated a reliable technique for selecting "great" software developers (or great scientists, or great engineers, &etc).
This problem has been examined again and again. The only way to find people with sufficient skill in any field is to test them (formal graded test, not idiosyncratic problems).
No one has ever demonstrated a reliable technique for selecting "great" software developers (or great scientists, or great engineers, &etc).
Exactly. It's akin to asking, "How can I tell that I've found Mrs. Right (or Mr. Right) during the first hour of talking to someone?"
I read about a study a few years back that required participants to spend some time with a member of whatever gender they're most attracted to. Each pair was required to (I think) do something slightly exciting and challenging together, talk about something significant, and look into each other's eyes a lot.
If I remember it correctly, several of the pairs from the study were engaged shortly thereafter.
So the answer to your question is simple, but not for the reasons you think.
If I remember it correctly, several of the pairs from the study were engaged shortly thereafter.
So the answer to your question is simple, but not for the reasons you think.
Would love a link to that if you can find it (assuming it exists somewhere on the interwebs).
I think its obvious someone fresh out of college will have "excellent algorithmic thinking" since that's how they spent the last 3/4 years. So in that case you should be spending more time during interviews looking at the more likely areas of weakness, which I'd sum up as "real-world experience." If you're interviewing experienced candidates, then you can generally hazard a guess that their areas of weakness are elsewhere (lack of passion, knowledge and experience that hasn't kept up with the state of the art, etc).
Interviews are all about probing. They shouldn't be a set of rote questions that you apply to every candidate regardless of background. You should make a guess at what the candidate's weaknesses are, and then push on them, push on them some more. This will also help you learn how well they deal with adversity.
Of course, before this, testing is a good way to determine what a candidate's strengths and weaknesses are.
As a total aside, I've found that US Customs and Border Patrol (the ones who do "secondary inspections") are about the best interviewers out there. They probe so much you end up feeling a little violated.
Interviews are all about probing. They shouldn't be a set of rote questions that you apply to every candidate regardless of background. You should make a guess at what the candidate's weaknesses are, and then push on them, push on them some more. This will also help you learn how well they deal with adversity.
Of course, before this, testing is a good way to determine what a candidate's strengths and weaknesses are.
As a total aside, I've found that US Customs and Border Patrol (the ones who do "secondary inspections") are about the best interviewers out there. They probe so much you end up feeling a little violated.
> You should make a guess at what the candidate's weaknesses are, and then push on them, push on them some more. This will also help you learn how well they deal with adversity.
I disagree.
Find out what they think that their relevant strengths are. This is usually fairly easy because they'll almost always tell you, often without being asked.
Test said strengths because that's where they'll go first, when things are tough, etc. Also, said strengths are probably where you want them working. If your needs are elsewhere, they're not your solution.
If their actual strengths are elsewhere, they've got a problem. Are they worth making said problem yours? (There are lots of reasons why their strengths might be elsewhere, some worse for you than others.)
I disagree.
Find out what they think that their relevant strengths are. This is usually fairly easy because they'll almost always tell you, often without being asked.
Test said strengths because that's where they'll go first, when things are tough, etc. Also, said strengths are probably where you want them working. If your needs are elsewhere, they're not your solution.
If their actual strengths are elsewhere, they've got a problem. Are they worth making said problem yours? (There are lots of reasons why their strengths might be elsewhere, some worse for you than others.)
I find that a 15-minute chat about a related topic that's not on their CV (i.e. something tangential to their "career") is a great way to sniff out hackers. Being a hacker myself, I have a very sensitive bullshit-o-meter and fairly varied interests, and I can quickly figure out whether that person is a "career programmer" who will end up disappointing, or someone who's got real passion for the job.
Add to that an intelligence filter (well, I have an affinity for smart people too), and maybe an informal chat about some past projects that they've worked on if still in doubt, and I think I can recognise good programmers even on a text chat system.
In fact, I spotted our last hire on IRC and was about 80% sure that he was someone we wanted to hire, after just 30 minutes of chatting.
Someone already posted the link to my article on how to recognise a good programmer - that breaks down the process I follow (though it's more internalised by now).
Add to that an intelligence filter (well, I have an affinity for smart people too), and maybe an informal chat about some past projects that they've worked on if still in doubt, and I think I can recognise good programmers even on a text chat system.
In fact, I spotted our last hire on IRC and was about 80% sure that he was someone we wanted to hire, after just 30 minutes of chatting.
Someone already posted the link to my article on how to recognise a good programmer - that breaks down the process I follow (though it's more internalised by now).
I'm not a business guy.
When hiring you should be thinking like a business guy.
You are dealing with a high level of cost (direct cost and opportunity cost), not to mention a massive degree of risk (people are not predictable).
Therefore it is worth some mitigiation - and the budget for it depends on your assessment of (risk * the total cost of a mistake).
The mitigations (such as a few months initially contracting for potential new hires, running internship programmes, recommendation bonuses) can then be costed and compared against the savings you make by reducing the probability of, or the cost of, a hiring mistake.
You are dealing with a high level of cost (direct cost and opportunity cost), not to mention a massive degree of risk (people are not predictable).
Therefore it is worth some mitigiation - and the budget for it depends on your assessment of (risk * the total cost of a mistake).
The mitigations (such as a few months initially contracting for potential new hires, running internship programmes, recommendation bonuses) can then be costed and compared against the savings you make by reducing the probability of, or the cost of, a hiring mistake.
Obvious answer: You probably can't identify the good ones without actually testing them.
I worked at an oil services company where I fairly regularly interviewed candidates. I was 180 degrees wrong on every interview I did. The bad ones turned out good, and the good ones turned out bad.
I would not, knowing what I know now, bother with anything else except a chat with some managers, and a some pseudo-code testing of the kinds of problems I'd expect a computer science grad to be able to solve.
This opinion is backed up by a basic training course for some new hires I gave recently, where my ranking basically flipped around from my first impressions, and the guys with actual real ability didn't really float to the top until after about 10 or 12 hours of training / evaluation. However, this could be shortened easily into an interview format.
There's a lot of people who hide their light under a bushel, and a lot of people who talk a good game, but are basically just warm bodies.
I worked at an oil services company where I fairly regularly interviewed candidates. I was 180 degrees wrong on every interview I did. The bad ones turned out good, and the good ones turned out bad.
I would not, knowing what I know now, bother with anything else except a chat with some managers, and a some pseudo-code testing of the kinds of problems I'd expect a computer science grad to be able to solve.
This opinion is backed up by a basic training course for some new hires I gave recently, where my ranking basically flipped around from my first impressions, and the guys with actual real ability didn't really float to the top until after about 10 or 12 hours of training / evaluation. However, this could be shortened easily into an interview format.
There's a lot of people who hide their light under a bushel, and a lot of people who talk a good game, but are basically just warm bodies.
I interviewed someone recently, someone much older than me, and it was a strange experience. We talked through his career, early on he had done some really interesting things, attacked some problems from unexpected angles and won big. But as we progressed through his career, it seemed to fade. Gradually all that early fire was crushed out of him and he became a 9-5er. So I thought, do I hire him and maybe try to turn him back into what he was? Or is it too late? In the end he was rejected by the other interviewer anyway, but it was a tough decision, and I could have forced it if I really believed in him. But it just goes to show, an interview can open up more questions about a person than it answers, and there's just no way to tell without actually working with them for some time.
You considered hiring someone to work 9-5 on your companies pet project as a way to break him out of his his dispirited 9-5 coder profile?
Oh, we'd have found something interesting and useful for him to do a couple of times, then we'd have expected him to identify such things for himself and just get on with it. Him 20 years ago would have loved it and contributed a lot - him now would have to either get it back, or sink without a trace.
I'd give a real world problem that is quite large, for example: Develop a school management system which synchronises across several computers.
I'd ask for 2-3 possible ways of designing the architecture. Then just drill down to the details, ask about the technology they would use, how they would go about implementing the algorithmic parts of things, the networking part of things, etc.
You'll get a good sense of how the person works on a high level as well as on a more detailed level. You'd also find out if the person is current on available technologies.
Algorithms are pointless to know, very rarely does one use that knowledge as a real life programmer. I know how quicksort works, but I've hardly ever _needed_ to know, and I've NEVER had to implement it.
I'd ask for 2-3 possible ways of designing the architecture. Then just drill down to the details, ask about the technology they would use, how they would go about implementing the algorithmic parts of things, the networking part of things, etc.
You'll get a good sense of how the person works on a high level as well as on a more detailed level. You'd also find out if the person is current on available technologies.
Algorithms are pointless to know, very rarely does one use that knowledge as a real life programmer. I know how quicksort works, but I've hardly ever _needed_ to know, and I've NEVER had to implement it.
They don't teach you the sorts in CS because you'll ever need to do them. It's merely a means to teach you Big-O by example. Being able to tell what's O(n) and what's O(n^2) is a very valuable real-life skill for a programmer.
I agree. Seeing how a programmer tackles a large problem is one of the best ways to gauge his skills/train of thought
I've seen people do great, well-commented code at interview, and then after I recommend them, they turn into mediocre 9-5 coders.
People always misrepresent at interviews; I can't think of a way around this.
People always misrepresent at interviews; I can't think of a way around this.
I ask questions about two things that seem great predictors. The first is interest in music. I'm not sure why but people who can read music and play any instrument well tend to make great programmers. I have no explanation for this. The second question is what the highest level math course they had was and what they found hard about it. This one reflects a personal bias. Knowing the answer for myself gives me a rough measure of how hard the person can think and their ability to handle abstractions.
> I'm not sure why but people who can read music and play any instrument well tend to make great programmers.
For what it's worth I came to exactly opposite conclusion, which probably means that this 'predictor' is a bogus one.
For what it's worth I came to exactly opposite conclusion, which probably means that this 'predictor' is a bogus one.
I would add, work together with them on a problem that is completely out of their comfort zone and/or background. See what kind of questions they ask, how fast they recognize a dead-end, and whether they learn from what you are trying to explain. If you have a hard time teaching them something new in an interview, probably it's not a good fit.
Try to give a (not too outlandish) problem that they cannot prepare for. People memorize algorithms or can adapt their mathematical knowledge, but when given an obscure or abstract scenario, you can get a good idea of how their mind works. Encourage them to take their time and think out loud so you can understand their process.
I was never a fan of contrived problems with arbitrary solutions. I agree that it offers a certain amount of insight into the interviewees thought process, but I tend to be a fan of problems with provably correct answers that require some rigor. Perhaps that's my problem?
I don't really mean problems that are unrealistic, just ones that are not everyday issues. Along with that, I think "provably correct answers" are mostly good, but it does not have to be just a single possible answer. I think the best problems are the ones with multiple good avenues, but you can still objectively identify if they take a wrong turn along the way.
I honestly think too much emphasis is placed on fixed questions and a particular style of interviewing. I've had a huge amount of success with improvised interviews that have both conversational elements drawn from the candidate's resume and some kind of problem solving task drawn from the type of work they will be doing.
I would probably give them a problem that involves processing/handling a large copious amount of data and compare their strategies.
Usually the great programmers come up with clean, elegant, efficient and scalable solutions.
Usually the great programmers come up with clean, elegant, efficient and scalable solutions.
Set up a competition geeks would be interested in, then try to hire the bests (for your definition of bests).
I'd not hire someone who had so few personal ideas to hack on, that he'd do every programming competition that came along.
What makes you think that the winners will want to work for you?
Many of the schemes for hiring great programmers assume that said great programmers are (1) looking and (2) want to work for you. Neither one is true. WRT (2), you have to sell them.
Many of the schemes for hiring great programmers assume that said great programmers are (1) looking and (2) want to work for you. Neither one is true. WRT (2), you have to sell them.
Setup a dozen seemingly random questions that will help you know what personality the person has.
Sit down for a week with each candidate, perhaps with all candidates, in a hackathon. Solving a real world problem. Then you can see who is worth what, their strong points, their weaknesses. It costs you time, but it's --in my experience-- worth every minute spent.
So how many week-long interviews are you planning to attend?
I don't doubt that this would work in a perfect world, but it's going to be difficult finding candidates willing to go through with it. Right off the bat you're going to have a severely limited pool of talent to choose from. It's also important to think about what kind of candidates would be up for this. They obviously wouldn't already have jobs and that could be for a variety of reasons - good or bad. These candidates would likely have to put off other interviews for a week, assuming they have other leads. Would you pay candidates during this interview process? That's going to have an effect on the types of people you can bring in for consideration. My guess is that in the end you're going to have just as many difficulties finding a suitable candidate.
I don't doubt that this would work in a perfect world, but it's going to be difficult finding candidates willing to go through with it. Right off the bat you're going to have a severely limited pool of talent to choose from. It's also important to think about what kind of candidates would be up for this. They obviously wouldn't already have jobs and that could be for a variety of reasons - good or bad. These candidates would likely have to put off other interviews for a week, assuming they have other leads. Would you pay candidates during this interview process? That's going to have an effect on the types of people you can bring in for consideration. My guess is that in the end you're going to have just as many difficulties finding a suitable candidate.
All the problems you bring up are real. Perhaps the hack-together approach could be rephrased as a screening stage, not a hiring stage. No point in blindly hiring people after testing them with some random, one-day stressful sessions.
(It's like scoring students with exams; makes no sense at all, when an informal oral exam would really tell you how much the student knows. Scoring students while having a beer with them may not be conventional, yet my former PhD advisor used that as a pre-screen for hiring new grads.)
(It's like scoring students with exams; makes no sense at all, when an informal oral exam would really tell you how much the student knows. Scoring students while having a beer with them may not be conventional, yet my former PhD advisor used that as a pre-screen for hiring new grads.)