Ask HN: Do you expect interviewees to “ace” whiteboard coding exercises?
16 comments
I used to (and, a few years ago, turned away smart people who failed them), but realized they were the wrong test for a small company. I switched initially to getting people to complete a task in their own time with the resources they liked and send us the code, and now entirely on network (i.e. who our CTO has already worked with and knows).
Big companies can't afford that luxury though. If big companies set tasks, the tasks would eventually leak because of the sheer number of people applying (both per position and over time). By necessity, you need the same task for all candidates to compare them effectively (ours was "build an inventory management system in Haskell for shoes", and we checked for things like documenting code, error management, and the sensible shortcuts someone might take), so you can't switch tasks around, and people will eventually learn what you like, work hard at faking the knowledge (e.g. copying and lightly modifying the code from other applicants), and you'll get loads of false positives.
So a big company needs to find relatively generalist questions that correlate roughly with later performance, and it seems that whiteboard CS questions are those by testing existing knowledge; through the sheer breadth of the exercise, it's a lot more time consuming to "revise" to fake that knowledge, and harder when on the spot to prove understanding. Are algorithm (and, less often mentioned, "fit") questions relevant to the job, maybe not, but you have to see the context and the hiring problem facing Microsoft or Google.
Of course, my opinion is that smaller companies asking whiteboard questions are just aping what they know from Google (as I was) and that there are better ways to hire when you have only 10 employees and 2 openings, but I might be wrong or at a local optimum.
I certainly regret not hiring a certain Masters in Statistics from Columbia who was very competent but who wasn't familiar with neural nets and admitted so in the interview/whiteboard... she'd have saved me at least 3 months on things I had to build later! Conversely, I later hired a guy who failed some of the whiteboard questions, but said it had caused him to learn more about the topic, which he did over a few months, at which point he came back to us, demonstrated competence and got an offer.
Big companies can't afford that luxury though. If big companies set tasks, the tasks would eventually leak because of the sheer number of people applying (both per position and over time). By necessity, you need the same task for all candidates to compare them effectively (ours was "build an inventory management system in Haskell for shoes", and we checked for things like documenting code, error management, and the sensible shortcuts someone might take), so you can't switch tasks around, and people will eventually learn what you like, work hard at faking the knowledge (e.g. copying and lightly modifying the code from other applicants), and you'll get loads of false positives.
So a big company needs to find relatively generalist questions that correlate roughly with later performance, and it seems that whiteboard CS questions are those by testing existing knowledge; through the sheer breadth of the exercise, it's a lot more time consuming to "revise" to fake that knowledge, and harder when on the spot to prove understanding. Are algorithm (and, less often mentioned, "fit") questions relevant to the job, maybe not, but you have to see the context and the hiring problem facing Microsoft or Google.
Of course, my opinion is that smaller companies asking whiteboard questions are just aping what they know from Google (as I was) and that there are better ways to hire when you have only 10 employees and 2 openings, but I might be wrong or at a local optimum.
I certainly regret not hiring a certain Masters in Statistics from Columbia who was very competent but who wasn't familiar with neural nets and admitted so in the interview/whiteboard... she'd have saved me at least 3 months on things I had to build later! Conversely, I later hired a guy who failed some of the whiteboard questions, but said it had caused him to learn more about the topic, which he did over a few months, at which point he came back to us, demonstrated competence and got an offer.
I agree with the work sample test. In similar threads, some found cheaters, but the followup interview exposed them because they can't speak about the process and solution cogently. I imagine that if you look at enough, just reading them will allow you to spot cheaters, and having so many applicants will provide a good selection ethical developers. And a short contract-to-hire or probation period should deal with any who get through.
The other issue with judging sample code is the sheer amount of time it takes. The two of us took around 3 months (full time, effectively, doing nothing else) to select, interview and make offers to around 10 employees out of a pool of over 100 qualified applicants.
You can sort of understand why companies try to develop less time consuming methods, and why experienced developers prefer to hire from their network...
You can sort of understand why companies try to develop less time consuming methods, and why experienced developers prefer to hire from their network...
Thanks for the data on the evaluation time. 9-10 hours per applicant is nontrivial, I agree. If you've worked with someone shoulder to shoulder, I can also see that being as effective. The problem I hear is that depending on the experience and number of the founding team, they can exhaust their network quickly.
Peopleware has the idea of auditions, and I thought if you were clear about what you wanted, applicants would be comparable. I'm thinking, "Walk us through a project you did that demonstrates your knowledge of code quality." Or similar, with a clear list of what you're hoping to see.
Peopleware has the idea of auditions, and I thought if you were clear about what you wanted, applicants would be comparable. I'm thinking, "Walk us through a project you did that demonstrates your knowledge of code quality." Or similar, with a clear list of what you're hoping to see.
Despite the sentiments of "they just want to see how you think" or "you don't have to get it right", in my experience if you don't do fairly well on the whiteboard you've failed the interview.
I think this partly depends on how hard the coding question is. If they give you FizzBuzz, I think you're pretty much required to ace it, even if you've never seen it. FizzBuzz is really simple though.
At Facebook I was getting I guess what you'd call classic algorithm questions but with an extra twist thrown in. If you combine that twist with excessive time restrictions (the Facebook interviews were strictly 45 minutes), you not only have to code well, you have to do it fast. On an algorithm that they appear to be hoping you've never seen before.
At Facebook I was getting I guess what you'd call classic algorithm questions but with an extra twist thrown in. If you combine that twist with excessive time restrictions (the Facebook interviews were strictly 45 minutes), you not only have to code well, you have to do it fast. On an algorithm that they appear to be hoping you've never seen before.
Yeah, that hope from them is baseless. Truth is, you're competing with candidates who have practiced a lot. They know it, and candidates know it too.
I'll re-iterate: Technical interviews are a competition. You cannot go unprepared. Testing your "raw" skill is stupid and a myth. It's as stupid as saying you want to see Usain Bolt (and everyone else) compete without practice. Those days are gone.
I'll re-iterate: Technical interviews are a competition. You cannot go unprepared. Testing your "raw" skill is stupid and a myth. It's as stupid as saying you want to see Usain Bolt (and everyone else) compete without practice. Those days are gone.
What they are describing is putting Usain Bolt at the beginning of an obstacle course and then saying he's not an Olympic level athlete when he trips on a tire.
What you're forgetting is that most developers, as with most athletes, aren't competing at the international level.
"Doing fairly well" to me is: you asked the right questions and converged on the solution, even if it took some time. Doing poorly means: you made stuff up instead of asking questions, you invented a convoluted way of solving the problem when some other data structure would have basically solved it for you, etc. It's also painful when the candidate's mental model of computation is miles away from reality.
Obviously, getting it right the first time would be great. But I'll settle for something not obviously insane.
Obviously, getting it right the first time would be great. But I'll settle for something not obviously insane.
Whiteboard or not, in any interview, you are judged relative to performance of other candidates. At very coveted places like FB, they don't have a dearth of candidates. i.e. Even if you did "good enough", very likely they'll find someone who "aced" the same question on the whiteboard. (We're not debating whether that's right judgment or not, but from their perspective, it's justifiable).
Having said that, I believe that your hunch is also correct viz. tech interviews for core roles at good companies these days, are more competitive than ever. Compared to say 10-15 years ago, the variety of interview problems is higher, complexity of software has increased, and the value of an engineer is understood to be higher ($Millions in equity). All that is contributing to moving of the goal post.
(Source: several years of doing this, now running an interview training course for a living http://InterviewKickstart.com).
Having said that, I believe that your hunch is also correct viz. tech interviews for core roles at good companies these days, are more competitive than ever. Compared to say 10-15 years ago, the variety of interview problems is higher, complexity of software has increased, and the value of an engineer is understood to be higher ($Millions in equity). All that is contributing to moving of the goal post.
(Source: several years of doing this, now running an interview training course for a living http://InterviewKickstart.com).
the value of an engineer is understood to be higher ($Millions in equity)
This would be at hip, happening startups in which you take a gamble on awful pay and terrible hours in the hope of being the unicorn, is it? Because in the vast majority of coding jobs, you're not worth millions.
This would be at hip, happening startups in which you take a gamble on awful pay and terrible hours in the hope of being the unicorn, is it? Because in the vast majority of coding jobs, you're not worth millions.
Value is always relative to supply and demand. Case in point: elementary school teachers are far more valuable, but they are still paid peanuts because there are so many of them.
Engineers who can work on large scale systems in the backend and frontend are not that many. And hence they are valued higher. (I'm not talking about commoditized skills like making webpages and configuring systems, which are indeed vast majority of software jobs).
How many hours you spend is a very good question, but it's a separate question from supply and demand (aka value) of core engineers.
Engineers who can work on large scale systems in the backend and frontend are not that many. And hence they are valued higher. (I'm not talking about commoditized skills like making webpages and configuring systems, which are indeed vast majority of software jobs).
How many hours you spend is a very good question, but it's a separate question from supply and demand (aka value) of core engineers.
Even at my small CRUD software company, there are 8 devs for a company bringing in $14million a year in revenue. That is certainly skirting the million range.
How many secretaries are there? By this reasoning, if there's one secretary, that secretary is worth millions.
And if they're any good, they probably are.
good point!
Care to share the company?
It really depends on the position and the problem. For entry development positions we will give simple tasks that can be 'aced' with any meaningful understanding of (web) development. Oftentimes more complex whiteboard exercises aren't even completed(they could take a long time, but used similarly to ensure problems are approached in an experienced manner.
For what it's worth, I think whiteboard coding exercises can be quite useful, if they are done right. And a complete waste of time if not.
Here "done right" means something like: used as a way to explore problem solving and logic with the candidate.
So in this light, I wouldn't expect anyone to "ace" a problem, but I would expect them to be able to say sensible things about whatever they try and to understand and discuss how to correct issues or alternatives if they are pointed out.
Here "done right" means something like: used as a way to explore problem solving and logic with the candidate.
So in this light, I wouldn't expect anyone to "ace" a problem, but I would expect them to be able to say sensible things about whatever they try and to understand and discuss how to correct issues or alternatives if they are pointed out.
I don't expect interviewees to "ace" whiteboard coding exercises, because the point of my questions isn't to answer the question: it's to see how you solve problems. Do you ask smart questions before writing anything? Do you put something down and then refine it? Do you handle errors? Do you understand the performance, maintenance, and testability implications of what you've written? That's what I'm looking for you to answer.
Then again, I work at Pandora, so this is probably an outdated approach...
Then again, I work at Pandora, so this is probably an outdated approach...
This isn't an outdated approach at all. It is how I conduct my coding interviews and how I have been interviewed in the past.
No, I don't do them at all. Stop telling people to code on the whiteboard for god's sake
I don't do whiteboard interviews but something similar (I'm in marketing). I'm not looking for people to ace the interviews, though obviously, it doesn't hurt. Instead, I'm looking for people who will be able to perform with training. They don't need to be perfect already and I don't expect them to be perfect. Instead, I expect candidates to have the basics down so that once they learn how we operate, they'll be able to perform.
My two cents: (not hiring for a well known west coast company, but still)
This is how I rank candidates. Let's say for a common question such as lowest common ancestor in a BST.
1. if they know the solution already and tell me they know it - major points, most interview prep books include it and they came prepared and they are also honest. I let them implement it still (and add some twist of my own)
2. If they don't know the solution and still figure it out, might even score higher, but people can "act" that they know it and some interviewed will fall for it. I want to believe I catch bad actors but can't say I can catch good ones. In any case if this is a question that appears in an interview prep book, I would expect them to know it (most people don't)
3. If they don't manage to implement it correctly, this is the biggest turn off. You CAN write correct whiteboard code if you WANT. I'm not with a stopwatch. If you don't then I consider it as sloppiness (more than "you can't code")
4. How you talk and behave - most important. If you can't figure it out yourself but ask me questions, if you try a simpler problem first, if you talk and explain what you are doing, if you run your ideas with me asking what I think, if you are determined to figure it out, ask for hints and immediately understand them and get this "aha" moment of how didn't I think of it rather than oh sucks I didn't think of it. If you are excited about the solution more than the fact I gave you hints for it, then I'd like to work with you. Asking questions is so important. Also not arguing with the interviewer and being a nice person really helps. I prefer shyness to overconfidence. Arrogance is an immediate no. Asking questions leads many times to a yes. Even if you didn't ace.
Bottom line with the existence of books courses and forums that list so many interview questions (some are on algorithms that took several years to develop) so knowing the solution is likely to be known from a book means you are not tested on your academic algorithmic development skills, you are tested on implementing something you read on CtCI with a twist perhaps. If anyone reads CtCI then interviews will just become harder.
This is how I rank candidates. Let's say for a common question such as lowest common ancestor in a BST.
1. if they know the solution already and tell me they know it - major points, most interview prep books include it and they came prepared and they are also honest. I let them implement it still (and add some twist of my own)
2. If they don't know the solution and still figure it out, might even score higher, but people can "act" that they know it and some interviewed will fall for it. I want to believe I catch bad actors but can't say I can catch good ones. In any case if this is a question that appears in an interview prep book, I would expect them to know it (most people don't)
3. If they don't manage to implement it correctly, this is the biggest turn off. You CAN write correct whiteboard code if you WANT. I'm not with a stopwatch. If you don't then I consider it as sloppiness (more than "you can't code")
4. How you talk and behave - most important. If you can't figure it out yourself but ask me questions, if you try a simpler problem first, if you talk and explain what you are doing, if you run your ideas with me asking what I think, if you are determined to figure it out, ask for hints and immediately understand them and get this "aha" moment of how didn't I think of it rather than oh sucks I didn't think of it. If you are excited about the solution more than the fact I gave you hints for it, then I'd like to work with you. Asking questions is so important. Also not arguing with the interviewer and being a nice person really helps. I prefer shyness to overconfidence. Arrogance is an immediate no. Asking questions leads many times to a yes. Even if you didn't ace.
Bottom line with the existence of books courses and forums that list so many interview questions (some are on algorithms that took several years to develop) so knowing the solution is likely to be known from a book means you are not tested on your academic algorithmic development skills, you are tested on implementing something you read on CtCI with a twist perhaps. If anyone reads CtCI then interviews will just become harder.
> f you can't figure it out yourself but ask me questions, if you try a simpler problem first, if you talk and explain what you are doing, if you run your ideas with me asking what I think, if you are determined to figure it out, ask for hints and immediately understand them and get this "aha" moment of how didn't I think of it rather than oh sucks I didn't think of it.
I don't understand what you mean with this sentence. Is this a good thing or not?
> If you are excited about the solution more than the fact I gave you hints for it,
Is this the best way? Gauging a person's emotional response? I'm pretty reserved in general and don't show emotions, but I do get fascinated internally by things, even if I don't intuitively show it. I just don't know how.
It feels like I'm being discriminated against because I'm not an extrovert cheerleader. Now that I know this, I can certainly act happy and excited, just for you.
But that's it: it's acting. I don't understand.
I don't understand what you mean with this sentence. Is this a good thing or not?
> If you are excited about the solution more than the fact I gave you hints for it,
Is this the best way? Gauging a person's emotional response? I'm pretty reserved in general and don't show emotions, but I do get fascinated internally by things, even if I don't intuitively show it. I just don't know how.
It feels like I'm being discriminated against because I'm not an extrovert cheerleader. Now that I know this, I can certainly act happy and excited, just for you.
But that's it: it's acting. I don't understand.
Some interviewers think they have innate abilities that even the best trained psychologists don't have. They are deluded. If you fail that interview you dodged a bullet (assuming you aren't 30 days from foreclosure or something).
I'm a little confused. You are saying you don't want people to be familiar with implementing an algorithm before you ask it? Like are there actually candidates who can come up with implementing red black tree rotation on the fly without ever having seen it, be it class or a book?
Can you clarify?
Can you clarify?
Well, depends on the question.
If it's a hard question, there are two options, if they know it already, great, I still would like to see it implemented. If not then I would like to see how they ask me questions and respond to hints and I'll help them get to the solution. I don't expect them to do years of algorithmic research on the fly, no. Example will be
If it's an easier question, I do expect them to come up with a solution on their own even if they never heard of it. Example will be a 2 sum question. Or rotations a matrix (just examples, I'm not saying that this is what I ask)
Hope I clarified.
If it's a hard question, there are two options, if they know it already, great, I still would like to see it implemented. If not then I would like to see how they ask me questions and respond to hints and I'll help them get to the solution. I don't expect them to do years of algorithmic research on the fly, no. Example will be
If it's an easier question, I do expect them to come up with a solution on their own even if they never heard of it. Example will be a 2 sum question. Or rotations a matrix (just examples, I'm not saying that this is what I ask)
Hope I clarified.
This is just my opinion, but If you're asking questions that can be found in "interview prep books" you're doing it wrong. Why not just ask questions that can be found by you know.. actually having experience developing software?
I do that too, but this type of questions serve two purposes. If they know it, I know that they do research before they approach a project, see what was done before, and take life seriously. Also it shows that they can learn something and actually implement it. Unless they memorized the solution, I can learn if they can actually code, not just talk.
Second if someone didn't know that question, it still shows how they think and behave.
I usually leave these to the end, but it's not always easy to come up with experience related questions that differentiate good candidates from great candidates.
Think of it this way, most of the day you don't need the skills tested in the interview, just like in college usually you don't need the stuff they test in SATs, right? You still need to find the right candidate somehow. Asking them about more practical / knowledge based questions is much harder. People come from different backgrounds. Also most people will know well basic software engineering questions. I'm left off with basic CS stuff (data structures and algorithms) to really find the difference point. My experience shows that great SEs will know to answer, not so great ones, won't. No explanation why. Perhaps great ones want to learn more than they need for their job. Good ones just do their jobs well. But when something new comes, or if they need to improve a slow algorithm, they give it up. Seen it happen many times to great "craftsmen" they can code CRUD and UI very well, but they can write an algorithm out of a paper bag. Sometimes this is exactly what I am looking for in a candidate. Depends on the role.
what type of work are you looking for?
Maybe I'm an edge case, but I have never once had to deal with a whiteboarding session in a single interview I have been in. Though, this is obviously a small sample. I've worked at one startup, largely contracted otherwise, and am just now looking to join another.
Largely my experience has been 1/2 to full day pair programming sessions, building a sample application that does something reasonably nontrivial, or a combination of these two.
I really personally enjoy this practice. Even though a day of billed hours is a large opportunity cost, so is the expense of potentially hiring a bad fit for the team. It goes both ways.
Largely my experience has been 1/2 to full day pair programming sessions, building a sample application that does something reasonably nontrivial, or a combination of these two.
I really personally enjoy this practice. Even though a day of billed hours is a large opportunity cost, so is the expense of potentially hiring a bad fit for the team. It goes both ways.
I don't understand how whiteboard algorithm challenges with a curveball or brain teasers are any indication of a developer's value add. Why would you not pair program with the candidate, and give them resources that are normally available to software developers? Memories are formed or helped by relationships (see mnemonics) and when you remove all context you're getting only a narrow view. You can "see how they think" by working with actual code.
It seems like the biggest players complaining about the lack of native tech talent in the US are the ones who give the worst tech interviews.
It seems like the biggest players complaining about the lack of native tech talent in the US are the ones who give the worst tech interviews.
Brain-teasers aren't overly helpful, IMO, unless they're directly related to stuff the company does.
For example, I had an odd one that addressed solution search space, how to short-cut going through the entire space, etc. Definitely a brain-teaser, and not something 99% of developers have likely dealt with--but relevant.
For example, I had an odd one that addressed solution search space, how to short-cut going through the entire space, etc. Definitely a brain-teaser, and not something 99% of developers have likely dealt with--but relevant.
1. Everyone will say they want to "see how you think"
2. You will not get an offer if you do not get a solution
3. Everyone ever: "White board interviews can be awful, but my company actually does a pretty good a job and I find them useful."
edit: for a slightly less snarky comment (and to expand on point 2) https://news.ycombinator.com/item?id=9522400
2. You will not get an offer if you do not get a solution
3. Everyone ever: "White board interviews can be awful, but my company actually does a pretty good a job and I find them useful."
edit: for a slightly less snarky comment (and to expand on point 2) https://news.ycombinator.com/item?id=9522400
I've gotten offers before where my answers to at least one of the whiteboard problems looked roughly like "I don't really know how to do this, but I'll see what progress I can make towards a solution" and either didn't result in a full solution or required pretty extensive help from the interviewer to do so. I've also seen offers made to people who didn't really get a good solution to one of the whiteboard problems posed when I was on the interviewing side but still did well in the rest of the interview and displayed good thinking and problem solving skills even in the question(s) they missed.
#2 is demonstrably wrong.
Of course there will be exceptions, but, given two candidates, the one who did better will get the offer. At big tech companies there is no shortage of people who ace a white-board problem.
Also, if you interview with multiple people for a position (and do multiple white-board problems) you can get an offer without acing all of them.
Also, if you interview with multiple people for a position (and do multiple white-board problems) you can get an offer without acing all of them.
Well yeah, but that's not what it said. I've had the occasional complete whiteboard disaster (not really my thing) that ended up with offers, including pools where people did bang-up whiteboard work.
There's a lot more at play in any reasonable place than whiteboard performance.
There's a lot more at play in any reasonable place than whiteboard performance.
> I know that whiteboard coding exercises are in low regard these days
I wouldn't say this. The hive mind holds them in low regard perhaps, but people hiring have to actually get the job done and in different companies different approaches are possible.
Why do you think it was the whiteboard? Perhaps you were just up against a really good candidate pool.
I wouldn't say this. The hive mind holds them in low regard perhaps, but people hiring have to actually get the job done and in different companies different approaches are possible.
Why do you think it was the whiteboard? Perhaps you were just up against a really good candidate pool.
[deleted]
I believe you should be able to use a computer because that's what they will be using day to day. Check out http://fizzbuzzer.com/. It allows you to send interview challenges to test a candidate.
Neat. I have used a few different tools in the past.
https://code.stypi.com/ - shared code editor in the browser
Just use hangouts (we are using https://appear.in/ now, much better) and have them screenshare with you, this will allow them to use their own editor and setup and make them feel more comfortable.
https://code.stypi.com/ - shared code editor in the browser
Just use hangouts (we are using https://appear.in/ now, much better) and have them screenshare with you, this will allow them to use their own editor and setup and make them feel more comfortable.
brettlangdon try http://fizzbuzzer.com/ and let me know what you think.
However, the last three whiteboard coding exercises I've done (two at Facebook and one at a small startup) seemed to be judged much more harshly than what I'm used too. I don't think I "aced" any of them, but had I been interviewing somebody that performed at the same level, I would have said "good enough" and passed the person along for further consideration.
Now some extra information: All three interviews were in Seattle, two of the interviewers for sure were ex-Amazon people (which I mention because I think it might be relevant), and I managed to get hired at Microsoft back in the 90s and at Google in the mid-2000s. I was not hired at Amazon back in 2002-ish, however.
So anyway, I'm wondering: Have I just not been doing as well as these interviews as I think, or has the industry moved the goal posts on me?