More whiteboard coding won’t build you a better team(blog.hackerhires.com)
blog.hackerhires.com
More whiteboard coding won’t build you a better team
http://blog.hackerhires.com/2012/09/more-whiteboard-coding-wont-build-you-a-better-team/
13 comments
> The worst have been ones where it was clear the interviewer had pulled questions out of the company's interview playbook and didn't have the depth of understanding to discuss them.
I'd argue that also belongs under the "best" list, because it told you exactly the level of the people you'd be working for/with.
I'd argue that also belongs under the "best" list, because it told you exactly the level of the people you'd be working for/with.
Very true.
In one such interview, I was asked for a way to implement Rock Paper Scissors in OOP without using a conditional. I came up with a neat way to do it with modulo operations and array lookups, but that wasn't The Right Way. So I came up with a way to do it with exceptions, but it wasn't The Right Way. Finally I came up with the solution the interviewer had in mind, which used Java's method overloading (my background is not Java).
I'm not sure if I "passed" the interview, but it certainly didn't make me want to.
In one such interview, I was asked for a way to implement Rock Paper Scissors in OOP without using a conditional. I came up with a neat way to do it with modulo operations and array lookups, but that wasn't The Right Way. So I came up with a way to do it with exceptions, but it wasn't The Right Way. Finally I came up with the solution the interviewer had in mind, which used Java's method overloading (my background is not Java).
I'm not sure if I "passed" the interview, but it certainly didn't make me want to.
Well, you basically showed that you don't understand OOP. That may not be important to you, but the interviewer probably successfully found out what he was looking for.
The most important aspect of OOP is message passing, and in Java, message-passing is implemented via non-final methods. OOP is a model of computation, just as FP and imperative programming are, and certain jobs require that you understand that model of computation well.
The most important aspect of OOP is message passing, and in Java, message-passing is implemented via non-final methods. OOP is a model of computation, just as FP and imperative programming are, and certain jobs require that you understand that model of computation well.
Well, if he eventually came up with the right solution, I would assume he understands OOP and overloading, as there usually isn't a way to just stop an interview and teach yourself core-foundations of a trade before answering questions.
I think the point was that, outside of specific cases, programming has many solutions. The fact that Paul gave two correct answers before getting to the 'right one' says that this was one of those cases.
Perhaps method overloading was not on the forefront of his mind that day.
A simple question of 'Why did you do it this way instead of using method overloading?' would've sufficed in figuring out whether or not they had a grasp on OOP and wouldn't have insulted the interviewee with open-ended questions where they were looking for one answer.
That would be the same as asking what is another way to write 49 and expecting only the squareroot of 2401.
I think the point was that, outside of specific cases, programming has many solutions. The fact that Paul gave two correct answers before getting to the 'right one' says that this was one of those cases.
Perhaps method overloading was not on the forefront of his mind that day.
A simple question of 'Why did you do it this way instead of using method overloading?' would've sufficed in figuring out whether or not they had a grasp on OOP and wouldn't have insulted the interviewee with open-ended questions where they were looking for one answer.
That would be the same as asking what is another way to write 49 and expecting only the squareroot of 2401.
But that highlights exactly the problem. He wasn't asked to demonstrate message passing in OOP, he was asked to implement Rock, Paper Scissors. There are numerous OOP-y solutions to that, but there's no way for him to know what they're actually looking for.
"Competant [sic] developers, even those who don’t specialize in JS, should be able to answer this question in 5-6 minutes. Therefore, this is a quick filter."
This is a false statement. I have programmed for years, but never in Javascript, so I have no clue what "add a cache to a Javascript function" even means. I can quite easily write code "that determines if a number is prime", but the assumption that everybody knows JS even if they don't "specialize" in it is problematic from my PoV.
This is a false statement. I have programmed for years, but never in Javascript, so I have no clue what "add a cache to a Javascript function" even means. I can quite easily write code "that determines if a number is prime", but the assumption that everybody knows JS even if they don't "specialize" in it is problematic from my PoV.
The question actually has nothing to do with JavaScript specifically, it's not much more than 'do you know what memoization is' if I understood it correctly.
I wonder if H1B hires have to go through this process. Does anyone go to India and ask people if they know how to cache an item in Javascript?
I don't care about these tests. I have a very simple interview process - I ask you what you've done and we talk through it. I may even ask you about how you've solved some problems involved with whatever you were working on. I then jokingly ask you about an inner join or a fib sequence and laugh after a few seconds.
Maybe it's because my I come from the family business of construction years ago, but I believe that it should be easy to hire someone and get them involved quickly. Then we can find out if they are worth it or not. Anything up until that point is talk, and talk is cheap. If they don't past muster eventually I get rid of them. Explaining this up front makes it even easier. I'll hire you quick, but if you don't do what I need you're out. It's not personal.
Maybe it's because my I come from the family business of construction years ago, but I believe that it should be easy to hire someone and get them involved quickly. Then we can find out if they are worth it or not. Anything up until that point is talk, and talk is cheap. If they don't past muster eventually I get rid of them. Explaining this up front makes it even easier. I'll hire you quick, but if you don't do what I need you're out. It's not personal.
Man... I don't know what I'd do if an interviewer presented that situation up front and very directly. "We'll hire you fast, but we won't hesitate to let you go." I wouldn't be able to stop thinking of the bad news bears scenario where I quit my job, get hired, and fired at the new one within a couple months. I guess it depends how awesome the job is that I'm applying for and how (not) awesome the job is that I'm leaving.
Try being on the country on a work visa. Makes that situation 100x more terrifying.
semi-serious devils advocacy. I don't know if the below is an argument that is correct but I hope for it to be proven wrong. I am pretty sure that accurately assessing individual capacity/potential is really, really, really hard.
"Google is a very successful technology company. While some of the problems they solve are very challenging and require innovative thinking and research, they have employees with decades of research experience and long publication track records to crack those.
New and seasoned developers will almost certainly have plug-n-chug everyday responsibilities, yet their interview process is still a fearsome battery of computer science questions. Their hiring also produces what is regarded as the most skilled CS workforce in the world and the market rewards them richly for it.
Why shouldn't we emulate this? Isn't it working? Aren't advocates of a disruption of this system essentially saying "Go ahead, relax your standards, what's the worst that could happen"? Is it taking your company 6 months to find a decent employee? Maybe that's about how long it should take because there just aren't that many decent employees."
"Google is a very successful technology company. While some of the problems they solve are very challenging and require innovative thinking and research, they have employees with decades of research experience and long publication track records to crack those.
New and seasoned developers will almost certainly have plug-n-chug everyday responsibilities, yet their interview process is still a fearsome battery of computer science questions. Their hiring also produces what is regarded as the most skilled CS workforce in the world and the market rewards them richly for it.
Why shouldn't we emulate this? Isn't it working? Aren't advocates of a disruption of this system essentially saying "Go ahead, relax your standards, what's the worst that could happen"? Is it taking your company 6 months to find a decent employee? Maybe that's about how long it should take because there just aren't that many decent employees."
The question is: what is your tolerance for (a) false negatives and (b) false positives?
Rephrasing (a): who are we passing on, are they in fact good potential employees, and what qualities do they have that our new hires might not?
Rephrasing (b): who have we hired that, despite whiteboard coding proficiency, isn't cutting it? And why?
In my experience whiteboard coding sessions feel adversarial, like a trap or a test, and you're basically waiting for the candidate to have an epiphany moment. Either they know the problem, or they don't but they get lucky and hit the "right" answer.
Doing reviews of code samples (or even pair programming against a new problem) puts interviewer and candidate both on the same side of the problem and feels more collaborative - I want to believe this gets you a better sense of what someone would be like to work with, and fewer false positives and negatives.
If anyone has evidence or studies either way I'd love to read them.
Rephrasing (a): who are we passing on, are they in fact good potential employees, and what qualities do they have that our new hires might not?
Rephrasing (b): who have we hired that, despite whiteboard coding proficiency, isn't cutting it? And why?
In my experience whiteboard coding sessions feel adversarial, like a trap or a test, and you're basically waiting for the candidate to have an epiphany moment. Either they know the problem, or they don't but they get lucky and hit the "right" answer.
Doing reviews of code samples (or even pair programming against a new problem) puts interviewer and candidate both on the same side of the problem and feels more collaborative - I want to believe this gets you a better sense of what someone would be like to work with, and fewer false positives and negatives.
If anyone has evidence or studies either way I'd love to read them.
I think this is the correct question to ask.
in my experience, false positives in early ventures (research projects, startups, project partnerships, whatever) are the kiss of death. If you need a group of people to do work at high capacity, bringing someone who does not 'fit' in when the group is young will probably destroy it. existing high-capacity people will become angry, there is less fault tolerance in the quality of the work and the rate at which it has to be done, etc.
it is usually the case that it is better to have 2 workers who click and are highly productive than 5 workers where 1 is not productive. so in the early days, failing closed seems very valuable.
when you are larger, there could be a higher tolerance. you could afford to hire someone and fire them a few months later if they do not work out, because the work that they are doing for you immediately is not as critical. however, there is also a certain culture that comes along with this and if you want to avoid that culture it seems like you still want to 'think lean' and part of that thinking could be to keep the amount of productive people high.
and it might also not go well with your current group members to say "yes, that hire was bad and we'll terminate them next week" for the fifth, sixth time in a quarter ...
in my experience, false positives in early ventures (research projects, startups, project partnerships, whatever) are the kiss of death. If you need a group of people to do work at high capacity, bringing someone who does not 'fit' in when the group is young will probably destroy it. existing high-capacity people will become angry, there is less fault tolerance in the quality of the work and the rate at which it has to be done, etc.
it is usually the case that it is better to have 2 workers who click and are highly productive than 5 workers where 1 is not productive. so in the early days, failing closed seems very valuable.
when you are larger, there could be a higher tolerance. you could afford to hire someone and fire them a few months later if they do not work out, because the work that they are doing for you immediately is not as critical. however, there is also a certain culture that comes along with this and if you want to avoid that culture it seems like you still want to 'think lean' and part of that thinking could be to keep the amount of productive people high.
and it might also not go well with your current group members to say "yes, that hire was bad and we'll terminate them next week" for the fifth, sixth time in a quarter ...
I agree this is the right question, but for the opposite argument. False positives in early stage ventures, in my experience, can be quickly and actively remedied. I've seen people fired less than two weeks after being hired without a substantial impact on the rest of the team or the product. If you can do this, you have a lot more freedom to take risks with new hires.
In a larger organization, if you hire someone, you're stuck with them in almost all cases. So be picky.
In a larger organization, if you hire someone, you're stuck with them in almost all cases. So be picky.
Rephrasing (b): who have we hired that, despite whiteboard coding proficiency, isn't cutting it? And why?
Has someone suggested whiteboard proficiency is sufficient by itself?
The whiteboard is meant to filter out people who suck. "Add a cache to a JavaScript function" is a dumb question for reasons many others have pointed out. Joel's classic article[1] recommends these as examples:
• Write a function that determines if a string starts with an upper-case letter A-Z
• Write a function that determines the area of a circle given the radius
• Add up all the values in an array
Those are all trivial. You should be able to come up with another half-dozen if you sit and think for a bit. The language doesn't matter, and the syntax shouldn't matter (as compared to the semantics).
I hear lots of scare-stories of "great coders who choke on interviews," but I find it very hard to believe any kind of "great coder" can't add up the values in an array. I might ask "how do you find the median value in the array" just to see what they come up with. There isn't a right answer, but there are lots wrong answers.
[1] http://www.joelonsoftware.com/articles/GuerrillaInterviewing...
Has someone suggested whiteboard proficiency is sufficient by itself?
The whiteboard is meant to filter out people who suck. "Add a cache to a JavaScript function" is a dumb question for reasons many others have pointed out. Joel's classic article[1] recommends these as examples:
• Write a function that determines if a string starts with an upper-case letter A-Z
• Write a function that determines the area of a circle given the radius
• Add up all the values in an array
Those are all trivial. You should be able to come up with another half-dozen if you sit and think for a bit. The language doesn't matter, and the syntax shouldn't matter (as compared to the semantics).
I hear lots of scare-stories of "great coders who choke on interviews," but I find it very hard to believe any kind of "great coder" can't add up the values in an array. I might ask "how do you find the median value in the array" just to see what they come up with. There isn't a right answer, but there are lots wrong answers.
[1] http://www.joelonsoftware.com/articles/GuerrillaInterviewing...
Has someone suggested whiteboard proficiency is sufficient by itself?
Fair point, that is a straw man. I haven't experienced an all-whiteboard interview - there's always been more to it.
The whiteboard is meant to filter out people who suck.
I think that's what some interviewers have missed. They jump straight in at the deep end with difficult algorithmic puzzles, and don't ever do an easy one. I have experienced this.
Fair point, that is a straw man. I haven't experienced an all-whiteboard interview - there's always been more to it.
The whiteboard is meant to filter out people who suck.
I think that's what some interviewers have missed. They jump straight in at the deep end with difficult algorithmic puzzles, and don't ever do an easy one. I have experienced this.
They jump straight in at the deep end with difficult algorithmic puzzles, and don't ever do an easy one. I have experienced this
Yeah, I have hard algorithm questions I ask, but I don't open with them, and I know they are hard. I expect them to possibly fill the entire remaining time.
Also, for the hard question, it's not even about getting it "right": the one person I ever had get my hardest question right was flushed by other people in my process, and I've hired many people who never got close. I want to see how they try.
Yeah, I have hard algorithm questions I ask, but I don't open with them, and I know they are hard. I expect them to possibly fill the entire remaining time.
Also, for the hard question, it's not even about getting it "right": the one person I ever had get my hardest question right was flushed by other people in my process, and I've hired many people who never got close. I want to see how they try.
It is adversarial, because it's a gating mechanism.
Sure, I'll be happy to discuss deep issues with you - after you've shown me that you're proficient in basic CS. It used to be that you could screen that over the phone, but now that you can look up the answers on your phone, that's kind of hard :)
Sure, I'll be happy to discuss deep issues with you - after you've shown me that you're proficient in basic CS. It used to be that you could screen that over the phone, but now that you can look up the answers on your phone, that's kind of hard :)
Assuming your whiteboard session is a good test of basic CS proficiency, the real question is "does basic CS proficiency correlate with effectiveness at the job?" - if you're like Google the answer is probably yes, and if so you only have to have tolerance for false negatives. Given enough applicants that's clearly less of a problem than false positives.
My other point is that some companies might find there are other qualities they're looking for that occur in the pool of people they pass on. That's what's really worth looking at, I think - all the while being careful not to mistake "good cultural fit" for "promotes long term monoculture".
My other point is that some companies might find there are other qualities they're looking for that occur in the pool of people they pass on. That's what's really worth looking at, I think - all the while being careful not to mistake "good cultural fit" for "promotes long term monoculture".
But the question he is raising with the false positives is that it is a gating mechanism for what? The ability to talk shop on a white board?
I'm assuming in my answer that the interview happens at a shop that cares about CS proficiency. (Mine certainly does ;)
If I have a false positive (i.e. you seem to understand CS but don't fit the company), it's not like the CS question is all that's asked. It's the launch point for a deeper discussion. That means a false positive hopefully gets caught in that later stage, but deep CS knowledge is sine qua non.
If that skill (knowledge of CS) doesn't matter to your shop, then yes, you shouldn't use it for gating.
If I have a false positive (i.e. you seem to understand CS but don't fit the company), it's not like the CS question is all that's asked. It's the launch point for a deeper discussion. That means a false positive hopefully gets caught in that later stage, but deep CS knowledge is sine qua non.
If that skill (knowledge of CS) doesn't matter to your shop, then yes, you shouldn't use it for gating.
Anecdotal, but I interviewed for a position where they gave me a tab deliminated data set, and said, "Show us what you can do with this in X language". I had two days.
It was one of my favorite interviews.
It was one of my favorite interviews.
Google hires the best because they have the capacity to pay the best. If you're hiring a mid-level developer in Texas and can only pay 70K a year, don't be surprised if your candidates have a hard time working out esoteric brain teasers.
I think this argument is simple, beautiful, and wrong. The assumption this seems to be built upon is that Google has developed a great hiring system that is doing well for us. I think that's a bit simplistic.
The reality is that (like everything at Google), the hiring process is constantly evolving and changing. Google puts considerable effort into improving its hiring process.
So, my counterargument is this: If Google is tweaking its hiring process and trying to make it better, why shouldn't you? Don't think of it as relaxing your standards, because it isn't. Think of it as changing your standards to better suit the kind of people you want to hire.
The reality is that (like everything at Google), the hiring process is constantly evolving and changing. Google puts considerable effort into improving its hiring process.
So, my counterargument is this: If Google is tweaking its hiring process and trying to make it better, why shouldn't you? Don't think of it as relaxing your standards, because it isn't. Think of it as changing your standards to better suit the kind of people you want to hire.
That question, and the example in the link really threw me for a loop.
At first I understood the question as described, but then looking at the javascript code I suddenly became lost and frightened. It wasn't until I clicked "run" that I realized the code wasn't working "as is" and needed to be made to work. Then it was just a matter of adding cache to the function (at least that's what I did). I had first read the blog as "this an example of caching" not "this is an example of the starting point for the problem".
I still prefer to not have 1 "golden" question but a list of problems the interviewee can choose from. That way they can find something that matches with areas they might be more famalier with (oh crap, I haven't thought of Fibonacci in awhile so I'll take on this robot-battle-engine quiz)
On an unrelated side note I had some fun making quiz #27 pass assertions but fail the meaning of the test:
At first I understood the question as described, but then looking at the javascript code I suddenly became lost and frightened. It wasn't until I clicked "run" that I realized the code wasn't working "as is" and needed to be made to work. Then it was just a matter of adding cache to the function (at least that's what I did). I had first read the blog as "this an example of caching" not "this is an example of the starting point for the problem".
I still prefer to not have 1 "golden" question but a list of problems the interviewee can choose from. That way they can find something that matches with areas they might be more famalier with (oh crap, I haven't thought of Fibonacci in awhile so I'll take on this robot-battle-engine quiz)
On an unrelated side note I had some fun making quiz #27 pass assertions but fail the meaning of the test:
for ( var i = 0; i < array.length; i++ ) {
var d = [];
d.fn = fn;
d.fn(i);
}"Advanced whiteboard coding" does have its uses:
1. It checks that you still know the basics of what's required for a CS degree.
2. Quite possibly, it also checks your capacity to (re-)learn some things for preparation.
Both of these are good indications of your ability to adapt to a new technical environment.
1. It checks that you still know the basics of what's required for a CS degree.
2. Quite possibly, it also checks your capacity to (re-)learn some things for preparation.
Both of these are good indications of your ability to adapt to a new technical environment.
I agree with the ability to learn part, but do you need the basics for what's required for a CS degree? I say this as someone who does not have a CS degree, but I also don't know that many places working directly with binary trees and linked lists, though I'll admit I may not have found them.
I've been doing software development for over ten years without a CS degree. Yeah, I've never had a problem that linked list solved, but I suspect having a CS degree might have better prepared me to solve certain classes of problems. After 10 years, I have a deep tool chest of patterns (and anti-patterns), but early in the game, having a better algorithmic eye would have certainly been beneficial.
The key thing is that 'you don't want to hire people to write code'. Code is inventory, code has a cost, code-not-written is what you want (as much as possible).
So you need to hire a developer who knows 'how not to code', 'when not to code', how to write as little code as possible (libraries, reuse and designing things that don't take much code, clean, concise, elegant, and well-structured).
These sort of developers are indeed the best to hire - but hard to find. And, as the article points out, is not what most technical interviews test for.
So you need to hire a developer who knows 'how not to code', 'when not to code', how to write as little code as possible (libraries, reuse and designing things that don't take much code, clean, concise, elegant, and well-structured).
These sort of developers are indeed the best to hire - but hard to find. And, as the article points out, is not what most technical interviews test for.
Even better, a developer that can take an existing code base and shrink it.
[deleted]
you mean get rid of it. Or, failing that, replace it with 30 lines of python.
Asking how someone would go about solving a problem is a good question to ask but simply can't replace making sure the candidate can write code. Imagine hiring script-kiddies instead of actual hackers.
As an aside, did anyone else see the irony in the word "competent" being misspelled?
As an aside, did anyone else see the irony in the word "competent" being misspelled?
Yep. We all did. Hence the "[sic]" used by some commenters in here when quoting the author -> "Competant [sic] developers, even those..."
On the other hand. What would be the difference between a script-kiddie and an "actual hacker"? The years of experience? Or the purpose for what the code will be used for?
On the other hand. What would be the difference between a script-kiddie and an "actual hacker"? The years of experience? Or the purpose for what the code will be used for?
I've been on both sides of the fence: being interviewed and interviewing. The times I've been interviewed I felt (most of the times) that either the person questioning me had no complete understand of what he/she was asking --it could be possible he/she was a junior engineer-- or it was a tricky algorithm question. The thing is that not everyone thinks or plans the same way. Analytical thinking is different for everybody. Most of the time I like to solve problems first either by brute force or the simplest solution and then try to improve it, but the author of the post also got a very good point: "why".
When asked questions without a context, you have no motivation and they seem weird. If you want to know if I know what a b-tree or a hash-table is, or what is database normalization, just ask me to explain you what it is. I also, wouldn't expect a candidate to implement one of those things right out of their head... that's just crazy. I would expect for him to be able to use them or at least how they work. If you need to see what I've done or how I perform against a particular issue your company might be having, code review of already code-reviewed code is a great, great way to test a candidate.
Github profiles are also a good indicator: it shows what you want, want motivates you, it can show real-world code that you have written.
Just talking of your experiences and how you've solved problems is also a good thing.
When asked questions without a context, you have no motivation and they seem weird. If you want to know if I know what a b-tree or a hash-table is, or what is database normalization, just ask me to explain you what it is. I also, wouldn't expect a candidate to implement one of those things right out of their head... that's just crazy. I would expect for him to be able to use them or at least how they work. If you need to see what I've done or how I perform against a particular issue your company might be having, code review of already code-reviewed code is a great, great way to test a candidate.
Github profiles are also a good indicator: it shows what you want, want motivates you, it can show real-world code that you have written.
Just talking of your experiences and how you've solved problems is also a good thing.
[deleted]
I've always thought that a good programmer job interview challenge would be to hand the candidate a laptop (preferably with their OS of choice) and have them build something simple but new. For example: "Have you ever built a web scraper in Ruby? No? Ok, Ruby's installed on this computer. Here's a CLI, a text editor, and a browser with Google open, feel free to make use of existing language resources, build me something that pulls some text off of www.example.com"
I think it's good to see how people learn new things and to see if the candidate looks for existing libraries rather than writing everything from scratch. You'd need to be able to give the candidate some time but I think it would be a valuable exercise.
I think it's good to see how people learn new things and to see if the candidate looks for existing libraries rather than writing everything from scratch. You'd need to be able to give the candidate some time but I think it would be a valuable exercise.
We used to do that at my previous company. We'd actually have several candidates come in together, have them form an ad-hoc team, and spend 6 hours on solving a real-world (toy) problem. With one existing team member on that test team, and a rotating team of observers.
It's a great method to hire, but it's expensive - you better make sure that the people you invite in are pre-checked quite well.
It definitely was the most fun I ever had interviewing.
It's a great method to hire, but it's expensive - you better make sure that the people you invite in are pre-checked quite well.
It definitely was the most fun I ever had interviewing.
[deleted]
Heresy!
Umm, polite sarcasm, anyone?
The fact of the matter is, the hypothesis that "whiteboard skills == mondo real life developer skillz" is pretty much an unspoken, and utterly unquestioned article of faith out in vast stretches of startupland.
The fact of the matter is, the hypothesis that "whiteboard skills == mondo real life developer skillz" is pretty much an unspoken, and utterly unquestioned article of faith out in vast stretches of startupland.
The worst have been ones where it was clear the interviewer had pulled questions out of the company's interview playbook and didn't have the depth of understanding to discuss them.