Ask HN: coding challenges as part of interviews: yes or no?
just curious what the group thinks of coding quizzes / challenges that are often part of job interviews for programmers. how predictive are they of a coder's future performance? are they appropriate only for entry level positions (when someone has less industry experience), or are they suitable for any level? are they more likely to pique your interest or to come across as cliche?
57 comments
My experience:
Successfully pasing a code challenge is not a good predictor of future success. However, failing to pass a code challenge is a very good predictor of future failure. Furthermore, making the test harder does not make it predict future success well.
Therefore, I belong to the "FizzBuzz" school: Make the test ridiculously simple. It should take no more than fifteen minutes to complete. If the person gets it obviously right, throw the result away and move on to more important quetsions. Do not try to deduce a lot of malarky about their programming style or ability from an obviously contrived problem.
For example, writing:
a = 5; b = 10
Does not mean they don't know Ruby well, and writing:
a, b = 5, 10
Does not mean they are a Ruby expert. And writing:
fsquare = lambda { |x| x * x }
Does not mean they are a closet Bipolar Lisp Programmer. If the program works, it's a pass, move on. If there is a syntax error or some such, who cares, move on.
But if they struggle... You need to investigate the reason for their difficulty with a simple problem.
Successfully pasing a code challenge is not a good predictor of future success. However, failing to pass a code challenge is a very good predictor of future failure. Furthermore, making the test harder does not make it predict future success well.
Therefore, I belong to the "FizzBuzz" school: Make the test ridiculously simple. It should take no more than fifteen minutes to complete. If the person gets it obviously right, throw the result away and move on to more important quetsions. Do not try to deduce a lot of malarky about their programming style or ability from an obviously contrived problem.
For example, writing:
a = 5; b = 10
Does not mean they don't know Ruby well, and writing:
a, b = 5, 10
Does not mean they are a Ruby expert. And writing:
fsquare = lambda { |x| x * x }
Does not mean they are a closet Bipolar Lisp Programmer. If the program works, it's a pass, move on. If there is a syntax error or some such, who cares, move on.
But if they struggle... You need to investigate the reason for their difficulty with a simple problem.
Trick questions, or questions with some kind of gottcha have no place in interviews, However, simple, straightforward programming exercises are essential. I've seen people who claim to have X years of industry experience who can talk a good game, but literally can't write a switch statement without reference to the internet.
I also like to ask people to peer review some code with both obvious and more subtle errors in it. That's a pretty important skill to me too. Anyway, ask them to code some fairly simple algorithm and then critique their own work. If there are several ways to solve the same problem, some more efficient than others, all the better.
I also like to ask people to peer review some code with both obvious and more subtle errors in it. That's a pretty important skill to me too. Anyway, ask them to code some fairly simple algorithm and then critique their own work. If there are several ways to solve the same problem, some more efficient than others, all the better.
Hmm... I know > 10 languages. I'm not sure I could remember the exact syntax in several of those languages on the spot. Is it case, is it switch, etc. If you do this, your test is on rote memorization.
Give them a novel problem and have them solve it with a real computer and in tact syntax lookup. Then see what and how they did it and go over their reasoning. I'll hire the guy with the wisdom over the guy who can spit out syntax trivia flawlessly. Those are different skill-sets that do not predict each other.
Give them a novel problem and have them solve it with a real computer and in tact syntax lookup. Then see what and how they did it and go over their reasoning. I'll hire the guy with the wisdom over the guy who can spit out syntax trivia flawlessly. Those are different skill-sets that do not predict each other.
Agreed. My first job they gave me a copy of Deitel & Deitel's "C How to program" along with the test. You actually lost points if you didn't open it!
You know the syntax of switch in the language you have been using in the last three months. If you don't, something's wrong!
I never use Ruby's case or switch or whatever-it's-called.
I could make up some mumbo-jumbo about case being a code smell and that it should actually be refactored to a method dispatch over polymorphic objects, but the truth is that I have no idea why I don't write case statements, I just don't.
I could make up some mumbo-jumbo about case being a code smell and that it should actually be refactored to a method dispatch over polymorphic objects, but the truth is that I have no idea why I don't write case statements, I just don't.
Add me to the "never uses switch statements" list. I've never seen the point in learning the syntax in any of the languages I know, since a bunch of else ifs (or elseifs, or elsifs, or elifs, depending on the language) does the same thing. If I waste a few keystrokes, but save a few brain cells, I think it's a worthwhile bargain.
Of course if I had to write that kind of construction often I'd probably learn the syntax of the case statement, but in the kind of work I do it hardly ever actually comes up.
Of course if I had to write that kind of construction often I'd probably learn the syntax of the case statement, but in the kind of work I do it hardly ever actually comes up.
It's really just a performance optimization for C, in situations where the switching value can be hashed. In languages that make method dispatch easy, and the overhead of a function call is not something you're terribly concerned about, it makes sense to just roll your own as needed -- that's why Python never bothered with a switch statement.
Actually, if you never use switch, then that's a symptom of being good at OO.
Sure, but we're talking about testing ability vs. memory - switch is incidental.
Ability requires memory, at least for basic things.
You can't be a good programmer and forget the syntax of the language you are working in. If you need to look up something as fundamental as switch statement in the internet, it's like a writer looking up words in the dictionary before writing them down in his novel.
You can't be a good programmer and forget the syntax of the language you are working in. If you need to look up something as fundamental as switch statement in the internet, it's like a writer looking up words in the dictionary before writing them down in his novel.
I think switch is a particularly bad example here. Honestly, I can't (accurately) tell you the last time I used one. It's not an every-day construct for a lot of people.
Neither Python nor Haskell have it - as far as I know. Perhaps you can create one out of monads though.
Actually, case is built into Haskell: it's the evaluation primitive in Haskell Core; see Don Stewart's blog post http://cgi.cse.unsw.edu.au/~dons/blog/2008/05/16 .
Yes, there it is. Sorry, I did not research my claim.
Actually pattern matching can also be described as something like a case-switch.
Actually pattern matching can also be described as something like a case-switch.
If you don't know the basic features of a language you don't know that language anymore. You maybe once knew it really well, and you still know the general syntax, but you aren't a insert language programmer anymore.
I do agree though that testing specific language skills is mostly pointless. The only time I've been given a programming test as part of an interview I was given a basic language spec that filled a couple pages then a book of problems to solve in the new language. Thats the way you should test programming skill in my opinion.
I do agree though that testing specific language skills is mostly pointless. The only time I've been given a programming test as part of an interview I was given a basic language spec that filled a couple pages then a book of problems to solve in the new language. Thats the way you should test programming skill in my opinion.
I don't think that I would ever hire a insert language programmer. When looking for someone to program, I'm just looking for a programmer; language isn't important.
A lot of people get hired in for emergencies. In such cases waiting for person to learn your offices language of choice isn't ideal.
Though they aren't always possible to avoid, I consider "emergencies" examples of poor business decisions and try to avoid them.
While I can see some merit in this argument, I'd also point out that I've never met a decent programmer that can't pick up a new language at least passingly in a day or two.
I agree wholeheartedly with the above. Gotcha questions are nice if you value having a sense of superiority over your candidates. Seeing how someone approaches a simple coding exercise tells you a lot about their techniques.
Two examples stand out:
1) A guy who had more Java certifications than had eaten hot dinners couldn't complete the simple exercise after three or four hours of effort.
2) Guy initially attempted to hit Google up for examples (why not!), and on finding that the net connection was disabled for that user account, wrote the best solution to the exercise I've received.
Overall, my interviewing process has had two stages. First, talk to the candidate and get a sense of what kind of person they are, and if they'll fit in with the culture. Second, if they seem reasonable, give them the test and see what their code is like. It's not foolproof, but it does weed out most of the weaker candidates.
Two examples stand out:
1) A guy who had more Java certifications than had eaten hot dinners couldn't complete the simple exercise after three or four hours of effort.
2) Guy initially attempted to hit Google up for examples (why not!), and on finding that the net connection was disabled for that user account, wrote the best solution to the exercise I've received.
Overall, my interviewing process has had two stages. First, talk to the candidate and get a sense of what kind of person they are, and if they'll fit in with the culture. Second, if they seem reasonable, give them the test and see what their code is like. It's not foolproof, but it does weed out most of the weaker candidates.
This is exactly how I was hired at my current employer. The VP of product dev interviewed me, and asked very general tech questions regarding work I've done. Then I was hammered by my now-direct manager. I didn't have any resources, but his questions often evaluated my thought processes as well as my coding skillz.
I should add that I thought I bombed the technical interview. So much so that I wanted to ask the VP if he had the right guy when he called with the offer.
I should add that I thought I bombed the technical interview. So much so that I wanted to ask the VP if he had the right guy when he called with the offer.
Some questions do let you guage how the person responds to stress and unknown environments - that may be helpful.
A trivial coding problem is essential. I've seen people with 10 years of experience in XML#Java.NETEnterprise2012 who don't understand conditionals, loops, or boolean expressions.
For phone screens, I find Steve Yegge's guide a useful starting point: http://steve.yegge.googlepages.com/five-essential-phone-scre...
I'm not a fan of asking candidates to reverse a string or to implement atoi or a binary search, because those have corner cases which, under interview pressure, might be too much to ask for. I just ask for an implementation of a gcd (greatest common divisor) function, in any language. I always explain the problem in detail. Only one guy (a PhD in physics) ever gave me Euclid's algorithm (section 1.1 in Knuth's AOCP Volume 1), so I don't expect it. I do expect a trivial loop, and most candidates take 15-20 minutes to come up with it, and need lots of hints.
As far as I'm concerned, anyone who can't come up with a trivial loop (or recursive or tail recursive function) for something like this shouldn't be programming for a living. Maybe "developing enterprise solutions," but never programming.
For phone screens, I find Steve Yegge's guide a useful starting point: http://steve.yegge.googlepages.com/five-essential-phone-scre...
I'm not a fan of asking candidates to reverse a string or to implement atoi or a binary search, because those have corner cases which, under interview pressure, might be too much to ask for. I just ask for an implementation of a gcd (greatest common divisor) function, in any language. I always explain the problem in detail. Only one guy (a PhD in physics) ever gave me Euclid's algorithm (section 1.1 in Knuth's AOCP Volume 1), so I don't expect it. I do expect a trivial loop, and most candidates take 15-20 minutes to come up with it, and need lots of hints.
As far as I'm concerned, anyone who can't come up with a trivial loop (or recursive or tail recursive function) for something like this shouldn't be programming for a living. Maybe "developing enterprise solutions," but never programming.
"As far as I'm concerned, anyone who can't come up with a trivial loop (or recursive or tail recursive function) for something like this shouldn't be programming for a living. Maybe "developing enterprise solutions," but never programming."
Priceless, If you ever put this on a cafepress mug or tee shirt, you can count on selling me a dozen or so as office gifts. Thanks for the laugh.
Priceless, If you ever put this on a cafepress mug or tee shirt, you can count on selling me a dozen or so as office gifts. Thanks for the laugh.
We use the (generalized) Euclidean Algorithm quite often in proofs. But - I study math.
This reminds me of a group of main frame programmers I knew who had been forced to become Perl web developers. About a month later they were asked to hire another Perl programmer and one of their interview questions was what's the difference between 'chomp' and 'chop' (chomp removes the final character of a string if it's a newline, chop does it regardless).
They were shocked that the interviewee didn't know the answer because they considered it basic knowledge, "it's in the first chapter for gosh sake." The first chapter in this case covered differences between Perl 4 and Perl 5.
In Perl 5, nobody uses 'chop', they all use 'chomp.' Many experienced Perl programmers might even have forgotten that 'chop' existed.
And that's my major problem with gotcha questions, they tend to be pretty far from measuring if a person can actually write productive code.
They were shocked that the interviewee didn't know the answer because they considered it basic knowledge, "it's in the first chapter for gosh sake." The first chapter in this case covered differences between Perl 4 and Perl 5.
In Perl 5, nobody uses 'chop', they all use 'chomp.' Many experienced Perl programmers might even have forgotten that 'chop' existed.
And that's my major problem with gotcha questions, they tend to be pretty far from measuring if a person can actually write productive code.
Programming challenges sort the wheat from the chaff - in particular those who cannot code at all, but beyond that I've rarely found them predictive of future performance. Much more important that you hire a good fit for your company. An unhappy or toxic coder can bring a startup to its knees - hire slow, fire fast and all that.
This is true. I aced the coding portion, but after 6 months, I was definitely not a good fit in the culture of the last company I worked for.
I like them as long as pseudo-code is acceptable. I don't really care how good someone's memory is with regards to language syntax; I care about whether or not they can solve problems and think them through.
/agree with most of the people here. Coding questions have a purpose.
You need to consider what you are trying to determine by asking any question. Really, what is it about the candidate you want to know? After pinning down precisely what you want to know then tailor a question to figure this out with as little peripheral stuff as possible.
If you want to know if someone can generate some code in their preferred language on the spot, fine. Pick a very simple problem, like 'Write a program to display numbers in the fibinocci sequence', and explain the fib seq to them if they don't know it. I wouldn't even count points off if they make any kind of mistake that a compiler would catch as long as they know how to fix it. Have them do it on the white board.
This kind of Q isn't designed to figure out how much they know of 10 different languages. It's designed to see if they can put some code down in their preferred language and have it do something useful.
The biggest constraint in an interview isn't the honesty of the candidate, or even their knowledge; it's the N hour time limit. Your goal as the interviewer is to squeeze as much information about who the candidate really is in the time allowed.
You need to consider what you are trying to determine by asking any question. Really, what is it about the candidate you want to know? After pinning down precisely what you want to know then tailor a question to figure this out with as little peripheral stuff as possible.
If you want to know if someone can generate some code in their preferred language on the spot, fine. Pick a very simple problem, like 'Write a program to display numbers in the fibinocci sequence', and explain the fib seq to them if they don't know it. I wouldn't even count points off if they make any kind of mistake that a compiler would catch as long as they know how to fix it. Have them do it on the white board.
This kind of Q isn't designed to figure out how much they know of 10 different languages. It's designed to see if they can put some code down in their preferred language and have it do something useful.
The biggest constraint in an interview isn't the honesty of the candidate, or even their knowledge; it's the N hour time limit. Your goal as the interviewer is to squeeze as much information about who the candidate really is in the time allowed.
I like to believe that I can get a pretty good understanding of someone's technical abilities by asking thorough questions about his previous experience, and maybe ask them to explain a couple of basic patterns or programming concepts to me. I remember interviewing a senior java developer who claimed 5+ years of experience and had no idea what a Singleton was. What made it worse was that even after explaining it to her, she was unable to figure out how to implement one.
Developers asked the strangest questions when interviewing other developers. I worked with a guy once who liked to ask his interviewees what the decimal value of 0xFE is. One candidate didn't know the answer so my coworker didn't think he could possibly be a good software engineer. I admit that (having done assembly coding in school) it seems odd that any software engineer wouldn't be able to figure out what FE is. But I still thought it was a bad way of testing technical skills. Would you rule out a candidate who doesn't know 0xFE ?
> Would you rule out a candidate who doesn't know 0xFE ?
If they didn't know it after a moment's thought, I'd cut them some slack; If they couldn't calculate it, with pencil and paper, in 10 minutes (and that's giving them plenty of extra time for interview-nervousness), then I'd be seriously concerned.
If they didn't know it after a moment's thought, I'd cut them some slack; If they couldn't calculate it, with pencil and paper, in 10 minutes (and that's giving them plenty of extra time for interview-nervousness), then I'd be seriously concerned.
i read through some answers here and i nodded in agreement with raganwald saying that this only worked as a filter on failure.
but then i read this and i thought "would i really want to give a job to someone who needs a pen and paper to subtract 1 from 255?"
i'm not sure what the lesson is here - perhaps that programming tests are dangerous because it is very easy to over-interpret the results, even when you start with the best of intentions?
[edited for grammar]
but then i read this and i thought "would i really want to give a job to someone who needs a pen and paper to subtract 1 from 255?"
i'm not sure what the lesson is here - perhaps that programming tests are dangerous because it is very easy to over-interpret the results, even when you start with the best of intentions?
[edited for grammar]
No, I would be very liberal in accounting for interview-nervousness.
Now, if it appeared that the candidate had no idea how to solve the problem, that would be a different story.
Now, if it appeared that the candidate had no idea how to solve the problem, that would be a different story.
Depends on the context. If this was a question posed for an embedded s/w eng position, I would not be inclined to hire.
I think taking some time to sit down and _read_ some code is one of the best and least used techniques during an interview. I think it's more of an indicator for job performance than writing some green-field code. After all, on the job, more time is spent reading and understanding code than writing code with no preexisting context.
You may use the solution your last candidate came up with as an example.
I think it depends on the position and responsibilities the person will have once hired. If the programmer will be coding directly from specs without any room for "interpretation" then a coding challenge may be very applicable. If you are looking for a programmer that also understands your business and will be required to make decisions and think on their feet, the value of a coding challenge diminishes.
It also depends on the time frame of the position. If you are looking to hire a programmer on a per project basis, then a solid understanding of the development language is critical. However, if you are hiring an developer for the long term, it's better to hire a smart and agile person that can contribute to the overall business as well -- which is hard to measure from a coding challenge.
It also depends on the time frame of the position. If you are looking to hire a programmer on a per project basis, then a solid understanding of the development language is critical. However, if you are hiring an developer for the long term, it's better to hire a smart and agile person that can contribute to the overall business as well -- which is hard to measure from a coding challenge.
I also worked for a company once who had several enthousiastic , seemingly bright people with somewhat limited experience come in for interviews, but they didn't make the cut because they didn't pass some coding challenges. The guy who did get hired went to a good school and was a good coder, but was way too arrogant, difficult to work with and repeatedly refused to do what his manager asked him to work on. And how did that guy interview other candidates? He screened them by their education, and when they would come in he would give them one programming challenge without asking any other questions. Nobody seemed to realize that there were big problems with the company's hiring strategies and that developers themselves were largely to blame for that.
I've always found that asking a reasonable numerical solving question where you can easily figure out the answer in your head is a good starting point to gauge programming/algorithm ability. If someone knows the solution they are trying to get at, they can focus on the program itself, as opposed to worrying about solving the problem and coming up with the algorithm. Ideally there are a few ways to solve the problem, brute-force methods or more efficient ways. It can be very difficult to test a programmer's ability in an interview, you need to focus more on seeing how the candidate thinks, how they approach problems, and whether they can admit when they do not know something.
I use a simple coding challenge as a weeder question. It requires basic use of recursion (without explicitly saying so) -- no math, no big-O, no gotchas, just writing a simple recursive function to process a small tree. It takes decent coders a few minutes to complete. Less-than-decent coders really struggle with it.
This is only a wheat/chaff separator, though. If possible, I like to see code from actual projects they've worked on to get a feel for the bigger picture (tip: open source your personal projects and obsess a bit over the code, so you have something to show off). Plus there's all the other non-code aspects to the interview.
This is only a wheat/chaff separator, though. If possible, I like to see code from actual projects they've worked on to get a feel for the bigger picture (tip: open source your personal projects and obsess a bit over the code, so you have something to show off). Plus there's all the other non-code aspects to the interview.
If you have the sense from whiteboard discussions that the candidate understands computer science (like gets recursion) but you want to make sure they can code, why not choose a subset of a problem that you (your company) needs to solve?
I do this, and provide a subset of REAL input data and perhaps a sample of the expected output. This ends up showing capabilities for more than just minor singleton operations, and shows that someone can use the atomic units of programming (loops, recursion, functions, etc) to actually produce a result. The chosen programming task is one someone in the office recently had to tackle themselves, so there is a working benchmark to compare against. I give it to them ahead of time, before an in-person interview, and say tell them I'm not necessarily expecting a working program (however, that would be nice), but come to the interview having considered the task and be prepared to talk about it, or talk about their specific solution/implementation. I find this also helps communicate the kinds of tasks the company deals with, and maybe a little of the company culture.
Good way to get some free consulting ;-)
I don't think he means open questions, but things related to the problem.
I interviewed once with a company that OCRed medical documents. One guy held up a sheet of paper and asked "How much memory does this take up?" which was a deliberately open-ended question. I was initially stumped, but then he started feeding me domain knowledge and assumptions, such as common DPI values and expected fidelity. Then I started coming up with estimates.
I don't think he expected me to have answer right off. I think he wanted to see how much I could figure out on my own if given hints.
I interviewed once with a company that OCRed medical documents. One guy held up a sheet of paper and asked "How much memory does this take up?" which was a deliberately open-ended question. I was initially stumped, but then he started feeding me domain knowledge and assumptions, such as common DPI values and expected fidelity. Then I started coming up with estimates.
I don't think he expected me to have answer right off. I think he wanted to see how much I could figure out on my own if given hints.
They're a necessity. For the simple reason that too many people lie on their CV, no matter how good their CV looks you have no-idea how much of it is accurate.
Obviously there are cases where you wouldn't use it, but generally they only apply when you know someone's capable of coding anyway (i.e. due to an indisputable reputation, experience working with them, etc.).
However as a predictive measure it's fairly limited in that it's a binary indicator. If someone fails then there's a fair chance they're a terrible developer, if someone passes there's a fair chance they're not a terrible developer.
Obviously there are cases where you wouldn't use it, but generally they only apply when you know someone's capable of coding anyway (i.e. due to an indisputable reputation, experience working with them, etc.).
However as a predictive measure it's fairly limited in that it's a binary indicator. If someone fails then there's a fair chance they're a terrible developer, if someone passes there's a fair chance they're not a terrible developer.
Necessary but not sufficient.
I always give a fairly simple problem, pencil and paper, a soda, and leave them alone for 15 to 30 minutes. Before I leave the room, I make sure they clearly understand what's expected.
What they do isn't important. Our discussion about what they did is. I don't care what syntax or mannerisms they use; I do care if they understand what was needed and what they did well enough to explain what they did and why they did it.
Almost every time some good follow-up questions and segues ensue. What about this? What about that? I've learned more about people in the discussion after the problem than from anything else.
I always give a fairly simple problem, pencil and paper, a soda, and leave them alone for 15 to 30 minutes. Before I leave the room, I make sure they clearly understand what's expected.
What they do isn't important. Our discussion about what they did is. I don't care what syntax or mannerisms they use; I do care if they understand what was needed and what they did well enough to explain what they did and why they did it.
Almost every time some good follow-up questions and segues ensue. What about this? What about that? I've learned more about people in the discussion after the problem than from anything else.
If I were interviewing someone, I would be more likely to test someone's general problem solving skills, questions that illustrate how a candidate thinks and approaches a challenge. This of course becomes more subjective, but can really help identify creative talent. Who knows, you might even learn something too.
This is probably a big part of the puzzle type questions that companies like Microsoft and Google seem to favor.
Agree with jamess, but additionally, I think interviewers that don't ask coding questions might be an indication of something negative at the company. Basic coding ability has to be one of those checkbox questions that determines your eligibility for the position, and if the interviewer doesn't ask anything technocal, it makes me wonder what kind of dumbasses I might end up working with.
Indeed, whether candidates write code during their interview is one of the items on the Joel Test (http://www.joelonsoftware.com/articles/fog0000000043.html).
Writing code is artwork.
You can ask the artist about the specifics of his tools, but you will never grasp his artistic ability until you see the completed artwork.
I just ask some general questions about the tools.
The tools for code writing are:
starting, constants, variables, addressing, comparing, branching and ending.
From the employees point of view: I would have no problem to do a coding test as part of an interview.