Don't Leak Interview Questions(igor.moomers.org)
igor.moomers.org
Don't Leak Interview Questions
http://igor.moomers.org/interview-questions/
28 comments
This post is based on two very common and very wrong assumptions: 1. To assess the candidate's competences you need some kind of magical, unique question that has only one right answer that reveals the competencies. 2. The interview is some kind of war/race between the recruiter and the candidate in which both sides have to outsmart each other. With such mindset companies loose the chance to recruit actual talent they need.
I do understand what AirBnb is trying to convey, but I sympathise more for the job seeker since I had really bad experiences with these types of question.
Once I was interviewing for a senior software engineer position for a big financial company. Before the start of the in-person interview process they had each candidate solve a Java programming question regarding encryption/decryption (according to their HR only 30% passes the initial filter and my solution was one of the best with regards to design, time to finish (I only spent 1hr to finish the exercise), runtime efficiency, and test coverage). They gave us each a laptop and 3 hours to solve the question. I got really good feedback after the exercise and was asked to go through the remaining gauntlet of more Java interview questions (2 more interview sessions w/ different people (tech lead and vp) to be exact).
At the last interview with one of the executive director, he dropped me a math puzzle question that is not related to the job at all, suffice to say I bombed the last interview. This was really frustrating and is a complete waste of time on my end since their office is a bit far from where I currently work.
Once I was interviewing for a senior software engineer position for a big financial company. Before the start of the in-person interview process they had each candidate solve a Java programming question regarding encryption/decryption (according to their HR only 30% passes the initial filter and my solution was one of the best with regards to design, time to finish (I only spent 1hr to finish the exercise), runtime efficiency, and test coverage). They gave us each a laptop and 3 hours to solve the question. I got really good feedback after the exercise and was asked to go through the remaining gauntlet of more Java interview questions (2 more interview sessions w/ different people (tech lead and vp) to be exact).
At the last interview with one of the executive director, he dropped me a math puzzle question that is not related to the job at all, suffice to say I bombed the last interview. This was really frustrating and is a complete waste of time on my end since their office is a bit far from where I currently work.
Always leak the interview questions. Always. Every time.
If you don't know how to pivot an interview question when the interviewee is already familiar with it, you aren't qualified to perform interviews.
If the goal is to find how they approach problem solving, you can adapt by asking about increasingly difficult aspects of the problem. If that isn't the goal, your hiring practices are wrong.
If the goal is to find how they approach problem solving, you can adapt by asking about increasingly difficult aspects of the problem. If that isn't the goal, your hiring practices are wrong.
While I appreciate the need for calibrated questions, it seems naive to expect a system to work if it depends on questions not to leak - especially if you're working with recruiters, or rejecting a lot of candidates.
What would be interesting is a system that works in spite of questions leaking. Personally I like the idea of a calibrated question pool so large that, if someone learned the answer to every question, they would have become qualified for the job. Of course you'd have to do a lot of recruiting to calibrate such a large question pool.
What would be interesting is a system that works in spite of questions leaking. Personally I like the idea of a calibrated question pool so large that, if someone learned the answer to every question, they would have become qualified for the job. Of course you'd have to do a lot of recruiting to calibrate such a large question pool.
[deleted]
This happens in consulting and engineering a lot. Case interviews and problems are known among fellow students who are interviewing later, and questions get put up on glassdoor too. You will have to iterate around the problem. This is unavoidable aspect. Shell Recruitment Day problems are known in detail online, their online assessment test is copied and solved in detail. Shell hires lot of engineers, and good ones. Consulting firms hold one of most intensive and competitive interviews. Questions will leak, issuing these moral statements will not help, this is your problem to solve.
I disagree. Good questions (and good interviews) don't require previous knowledge (and if it exists, this can be worked around, or will have little effect)
You know, it's not about a, b, c or d. It's the whole answer.
And I still don't know why interviewers insist on whiteboard coding. Get a computer, google stuff together, try some code, because that's how it works in the end!
You know, it's not about a, b, c or d. It's the whole answer.
And I still don't know why interviewers insist on whiteboard coding. Get a computer, google stuff together, try some code, because that's how it works in the end!
>And I still don't know why interviewers insist on whiteboard coding.
Worse. They require syntax correct whiteboard coding. I got to imagine that pseudo-code should suffice in almost every case.
Worse. They require syntax correct whiteboard coding. I got to imagine that pseudo-code should suffice in almost every case.
did you even read the original post? we talk about a question on a whiteboard, and then move to solving it on a computer. i mentioned several times that we try to get to real running code.
Yes, I read it. The point about whiteboard coding is generic, and not relating specifically to the article.
Related by possibility off-topic: How about signing a NDA?
I am not advocating it (in fact, I probably would have significant reservations about any company that will pull that on me), but if you are that concerned about the questions leaking out, do something to legally oblige the candidate not to disclose the process.
Otherwise, put up with it. Or try a different way to vet out candidates.
I am not advocating it (in fact, I probably would have significant reservations about any company that will pull that on me), but if you are that concerned about the questions leaking out, do something to legally oblige the candidate not to disclose the process.
Otherwise, put up with it. Or try a different way to vet out candidates.
GOOG, FB makes you sign an NDA when you interview, but this can't and doesn't stop you from posting on glassdoor.com anonymously.
Also, I might add, some people like to post their interview questions to see what other people may think or get feedback on their answer. Because one thing that you don't get when you intw, especially if you don't get the job, is any feedback on your interview performance (for myriad of legal reasons).
Also, I might add, some people like to post their interview questions to see what other people may think or get feedback on their answer. Because one thing that you don't get when you intw, especially if you don't get the job, is any feedback on your interview performance (for myriad of legal reasons).
I had to sign a NDA at a tech interview once, and they asked me to code FizzBuzz.
That should be a t-shirt slogan.
[deleted]
You do realize you've just violated that NDA, right?
Consider:
If your interview process can be substantially weakened by having some of your questions leaked... then maybe it wasn't such an effective process in the first place.
If your interview process can be substantially weakened by having some of your questions leaked... then maybe it wasn't such an effective process in the first place.
Cry me a river AirBnb.
Why not just give the candidate a programming problem, a real computer (maybe no internet, but all tools and documentation installed), and an hour or two to solve the problem?
You can monitor their computer screen remotely, record it even for later review, and also sit in the room to watch them at work and offer guidance if necessary.
If the candidate passes these real world challenges, then you can do the soft social stuff and see if they will fit into the team.
Why not just give the candidate a programming problem, a real computer (maybe no internet, but all tools and documentation installed), and an hour or two to solve the problem?
You can monitor their computer screen remotely, record it even for later review, and also sit in the room to watch them at work and offer guidance if necessary.
If the candidate passes these real world challenges, then you can do the soft social stuff and see if they will fit into the team.
totally agree. actual problem solving is more informative than questioning.
but even for questions regarding prior experience - most of the "standard" job interview questions are widely known. the question being widely known doesn't make them less effective - it is up to the skill of the interviewer to ask relevant follow ups based on what the candidate says.
often, I don't even ask standard" questions. i just look at the most audacious claim on the resume and grill on that topic to figure out what they actually did.
Then i jump around from claim to claim, out of sequence, so they can't fall into the rote story of "how I got here."
I think this is a reasonably effective, harder-to-hack way to interview, unless you're dealing with a pathological liar. (so far, I haven't personally hit one.)
but even for questions regarding prior experience - most of the "standard" job interview questions are widely known. the question being widely known doesn't make them less effective - it is up to the skill of the interviewer to ask relevant follow ups based on what the candidate says.
often, I don't even ask standard" questions. i just look at the most audacious claim on the resume and grill on that topic to figure out what they actually did.
Then i jump around from claim to claim, out of sequence, so they can't fall into the rote story of "how I got here."
I think this is a reasonably effective, harder-to-hack way to interview, unless you're dealing with a pathological liar. (so far, I haven't personally hit one.)
> ...unless you're dealing with a pathological liar. (so far, I haven't personally hit one.)
Or he was really good. :)
Or he was really good. :)
Interesting.
Do you do this with all hires, or just junior candidates e.g. recent graduates, those having taken bootcamp classes, or migrated over from another discipline e.g. "I used to be in Marketing but now I'm a growth hacker" ?
What if you have a candidate with a proven track record, perhaps lots of open source code, perhaps well known in the technology community? Isn't there a risk that grilling them on a topic might just antagonize them to the point of not wanting to join the company?
Do you do this with all hires, or just junior candidates e.g. recent graduates, those having taken bootcamp classes, or migrated over from another discipline e.g. "I used to be in Marketing but now I'm a growth hacker" ?
What if you have a candidate with a proven track record, perhaps lots of open source code, perhaps well known in the technology community? Isn't there a risk that grilling them on a topic might just antagonize them to the point of not wanting to join the company?
good question. if you have first hand knowledge of a candidate's qualifications, it is going to change the interview for sure. but that doesn't mean you don't need to figure out if there is a fit with your team/company.
i don't think focusing on specific details is antagonizing if it is done right.
a great candidate will embrace the opportunity to talk about their experience. it will probably be an awesome conversation for both of us. i will learn something, and they will see their experience in a new light trying to explain to someone else.
a weak candidate will stumble on details of their experience, or will be forced to admit someone else did steps x, y, or z. or that the claim is inflated.
i don't think focusing on specific details is antagonizing if it is done right.
a great candidate will embrace the opportunity to talk about their experience. it will probably be an awesome conversation for both of us. i will learn something, and they will see their experience in a new light trying to explain to someone else.
a weak candidate will stumble on details of their experience, or will be forced to admit someone else did steps x, y, or z. or that the claim is inflated.
I couldn't agree more. My favourite questions are along the lines of
"What technology are you most experienced with? Can you explain it's architecture to me?" and
"What is the most complex customization you did to an existing library/framework? Can you explain how (and why) you did it?"
They're completely scripted questions but tell you much more about the candidate than doing an algorithm pop-quiz.
"What technology are you most experienced with? Can you explain it's architecture to me?" and
"What is the most complex customization you did to an existing library/framework? Can you explain how (and why) you did it?"
They're completely scripted questions but tell you much more about the candidate than doing an algorithm pop-quiz.
It's amazing so many interviewers can't apply good problem solving skills to their own job, which of course is to test problem solving skills :)
It's almost as if they want to suffer and fail. We spent a decade hiring puzzle solvers, only to announce (few years back) that it's useless and must be discontinued asap. Now let's try whiteboarding classic logic problems to see if you will catch an off-by-one mistake without running it. That will work.
It's almost as if they want to suffer and fail. We spent a decade hiring puzzle solvers, only to announce (few years back) that it's useless and must be discontinued asap. Now let's try whiteboarding classic logic problems to see if you will catch an off-by-one mistake without running it. That will work.
Your "solution" doesn't address the problem in any way. That method of testing requires just as much preparatory work before it can be used in a live interview with any confidence. It's also no less vulnerable to the question being leaked.
Playing the world's saddest song on the world's smallest violin.
Off-topic: The person's thumbnail photo is a few megapixels in size and almost 50 kb :(
If your interview process relies on simple riddle questions, you might want to reassess the way you interview. The most constructive interviews I've been in involve pair programming and architectural discussion. Focus less on questions and more on seeing if this candidate can do the things you want them to.
OTOH, someone who seeks out interview coding problems and solves them might be a good fit for your team. If they're just memorizing answers someone else came up with I agree that's misleading, but I would also imagine they would not be able to talk about it intelligently/insightfully in the interview.
Folks -- the value in this article is not in the admonition about leaking questions (the HN title does not help). The value is that the author is spot-on about how we (I include myself) should approach the task of conducting the technical interview. Read the article again; leave out the last section if it bothers you that much.
Considering how stacked the interview is against the candidate, I really have no empathy for the interviewers in situations like this.
Throwing puzzlers at a candidate indicates you are hiring for normal engineering talent (i.e. this person doesn't have a previous reputation that makes them special) and so vetting the person is going to be highly imperfect whether or not they can solve your puzzle.
Throwing puzzlers at a candidate indicates you are hiring for normal engineering talent (i.e. this person doesn't have a previous reputation that makes them special) and so vetting the person is going to be highly imperfect whether or not they can solve your puzzle.
I have signed NDAs in which I have agreed not to disclose interview questions I was asked, and I think NDAs are a fair solution to this problem. The only potential problem for an organization like Airbnb is that it can require a lot of infrastructure and legal horsepower, which perhaps only larger companies have the luxury of.
How could such an NDA be enforced? You generally wouldn't have any way of proving who leaked the question (unless you ask every candidate a unique question), and even if you did spend the time and expense to litigate against the leaker, it would be hard to convince a jury that disclosing the question would have incurred a non-trivial loss for the company.
If an interviewer asked me to sign an NDA, I'd think the company was run by a bunch of crazy control-freaks and walk out.
If an interviewer asked me to sign an NDA, I'd think the company was run by a bunch of crazy control-freaks and walk out.
This is somewhat standard practice. At Google/Facebook campuses you have to agree to an NDA in order to print out your guest badge.
When you publicly post a question that you were asked after interviewing, you are undoing weeks of work for your fellow engineers. In it's place, you a creating weeks of new work as we develop, test, and calibrate on new questions.
There's something very broken about a process which takes weeks to prepare a question, and something even more broken when programmers conduct interviews up to 6 times a week (how do they have time for any real work?). That's more than 1 interview a day...
Questions shouldn't be secret puzzles which take ages to fabricate, but ways for the candidate to demonstrate what they know and what they can do. Otherwise you're leaving wide open the possibility that you'll hire someone good at gaming the system, not someone good at the job.
There's something very broken about a process which takes weeks to prepare a question, and something even more broken when programmers conduct interviews up to 6 times a week (how do they have time for any real work?). That's more than 1 interview a day...
Questions shouldn't be secret puzzles which take ages to fabricate, but ways for the candidate to demonstrate what they know and what they can do. Otherwise you're leaving wide open the possibility that you'll hire someone good at gaming the system, not someone good at the job.
You know at this point, a candidate who had prepped on known questions would be a huge plus in my book. Unbelievable how many candidates I interview that don't even know a single thing about the product, and who are unable to code the completely predictable problems you know will be thrown at you in a dev interview. Did they just wake up from a drunken stupor in the lobby with a note pinned to their chest I wonder? It boggles the mind.
You know at this point, a candidate who had prepped on known questions would be a huge plus in my book.
That's what I'm thinking. Honestly... how many unqualified prospects are going to try to gain a position by memorizing FizzBuzz solutions, or whatever? Is this an actual problem in the real world? Wouldn't it be easier to just learn to code from first principles?
That's what I'm thinking. Honestly... how many unqualified prospects are going to try to gain a position by memorizing FizzBuzz solutions, or whatever? Is this an actual problem in the real world? Wouldn't it be easier to just learn to code from first principles?
Phone screen with FizzBuzz. Admit that it's not complicated, but that you get so many unqualified people that it does serve to filter, but make them do it anyways. It's 5 minutes at most and saves them the effort of traveling to your office AND saves you the effort of interviewing seriously.
My comments were actually applying to the phone screen portion of the interview. But it's still an hour of everyone's time.
The negative comments in this thread seem representative of bad personal experiences and not the article.
He's saying that he wants a chance to work through a problem with the interviewee, not stump them. There is value to seeing what a candidate knows and has previously done, and I'm sure that by this point in the interview he has determined those.
But these questions are about how the candidate works, and it sounds to me like he has put a lot of effort into making sure this is not like a crappy puzzler, but a conversation that he will help them through.
He's saying that he wants a chance to work through a problem with the interviewee, not stump them. There is value to seeing what a candidate knows and has previously done, and I'm sure that by this point in the interview he has determined those.
But these questions are about how the candidate works, and it sounds to me like he has put a lot of effort into making sure this is not like a crappy puzzler, but a conversation that he will help them through.
I think the better option is to ask questions that don't have one correct answer? Those are the hardest ones anyway, and the most relevant to the real world.
You explore the options beforehand, and try to figure out benefits / drawbacks. Then when you ask the question, you see if people see those tradeoffs too. Even better, you pose the question slightly differently each time, such that a different option is more favored every time. This filters out people who think they have the "right" answer.
Who knows - one day a candidate could even surprise you with an answer you didn't think of!
You explore the options beforehand, and try to figure out benefits / drawbacks. Then when you ask the question, you see if people see those tradeoffs too. Even better, you pose the question slightly differently each time, such that a different option is more favored every time. This filters out people who think they have the "right" answer.
Who knows - one day a candidate could even surprise you with an answer you didn't think of!
I have a lot of the same qualms shared by many of the responses so far in regard to how unrepresentative to real world work the typical whiteboard coding interview is.
But even if I didn't, what's the point of asking such a large group of people not to leak questions, especially when those most put off by your interview style (for whatever reason) have the most motive to leak your questions?
At best you'll be ignored, laughed at a bit and people doing what you're asking them not to do will continue doing it. At worst (and IMO the more likely outcome) you'll end up with a variant of the Streisand effect where even more people leak just because you asked everyone not to.
In closing, I guess I will mention how odd it seems to me that "startups" (granted, Airbnb being a startup anymore is debatable) are all "pivot this, disrupt this, change the system", and then complain when an interview style that was already old and busted in 1970s isn't working out for them in this new Internet world.
But even if I didn't, what's the point of asking such a large group of people not to leak questions, especially when those most put off by your interview style (for whatever reason) have the most motive to leak your questions?
At best you'll be ignored, laughed at a bit and people doing what you're asking them not to do will continue doing it. At worst (and IMO the more likely outcome) you'll end up with a variant of the Streisand effect where even more people leak just because you asked everyone not to.
In closing, I guess I will mention how odd it seems to me that "startups" (granted, Airbnb being a startup anymore is debatable) are all "pivot this, disrupt this, change the system", and then complain when an interview style that was already old and busted in 1970s isn't working out for them in this new Internet world.
I have no sympathy here - interviewing is hard for both parties and isn't going to be solved by imposing rules like this; you won't get better hires by asking them not to prepare.
Also, (possibly off topic...) am I the only one that thought of the Kobayashi Maru after reading this?
Also, (possibly off topic...) am I the only one that thought of the Kobayashi Maru after reading this?
In the list of immoral things, leaking interview questions comes much below spamming and creating fake profiles.
True enough, but I'm not sure I see the relevance to the point of the article?
He's talking about http://davegooden.com/2011/05/how-airbnb-became-a-billion-do...
I'm European and have rarely had to be interviewed, but the few interviews I've been on have never asked me any programming specific questions, rather focusing on what my interests are and how I am as a person.
It sounds to me like many companies are trying to hire quiz solvers and people who can figure out how many people can fit into a car. Do the people being interviewed not have any downloadable code samples or coding projects that the company could have eyed through prior to the interview?
It sounds to me like many companies are trying to hire quiz solvers and people who can figure out how many people can fit into a car. Do the people being interviewed not have any downloadable code samples or coding projects that the company could have eyed through prior to the interview?
No. You can pretty much go fuck yourself.
I hate interviewing as it current exists and I'm going to make sure each and every one of my friends know what I got asked so that they can better prepare themselves and have a better experience.
I hate interviewing as it current exists and I'm going to make sure each and every one of my friends know what I got asked so that they can better prepare themselves and have a better experience.
Quite honestly, and I have been on both sides here, my sympathy goes rather to the job seeker who is trying to find a better job in one of very few industries where you have to solve puzzles to get a job!
Not to mention tech. giants that ask inhuman questions just because everyone wishes to work for them.
Sorry, until things are different, I'll leak every interview question I get my hands on. Tech. companies will always be able to figure out ways to examine candidates.
Not to mention tech. giants that ask inhuman questions just because everyone wishes to work for them.
Sorry, until things are different, I'll leak every interview question I get my hands on. Tech. companies will always be able to figure out ways to examine candidates.
I agree: I won't leak your interview questions if you'll agree to make them fair. Deal?
To me, fair means: not asking me to write code on a whiteboard! Or, if you insist upon it, not complaining about not-compiler-perfect syntax, assisting me with the correct function names or argument order of library functions I'd normally autocomplete or look up, if I've forgotten, and focusing on the thought process and problem-solving rather than keypresses.
But really, most fair would be: don't ask me to code your stupid whiteboard interview question. Look at the shitloads of code I have on github, or ask me to walk through some interesting problems I had to solve recently, or ask me for my first-blush take on an interesting problem you're trying to solve. Make it a conversation instead of an interrogation.
Because, you know, a startup filled with people who are excellent at solving interview questions on a whiteboard might not be the best team for solving real-world problems. I work with a guy who can solve a rubik's cube in under 90 seconds -- that's about as relevant as some interview questions I've seen.
To me, fair means: not asking me to write code on a whiteboard! Or, if you insist upon it, not complaining about not-compiler-perfect syntax, assisting me with the correct function names or argument order of library functions I'd normally autocomplete or look up, if I've forgotten, and focusing on the thought process and problem-solving rather than keypresses.
But really, most fair would be: don't ask me to code your stupid whiteboard interview question. Look at the shitloads of code I have on github, or ask me to walk through some interesting problems I had to solve recently, or ask me for my first-blush take on an interesting problem you're trying to solve. Make it a conversation instead of an interrogation.
Because, you know, a startup filled with people who are excellent at solving interview questions on a whiteboard might not be the best team for solving real-world problems. I work with a guy who can solve a rubik's cube in under 90 seconds -- that's about as relevant as some interview questions I've seen.
Who does this? Seriously? I've been on probably a dozen interviews where I would rank my performance from dismal to awesome, gotten offers. I was never, under any circumstance, expected to write syntax perfect code on a whiteboard. I have written plenty of stuff on whiteboards, in a few cases until my hand was cramped, but never did anyone bitch about the kind of issues you mention here.
A good white board interview question asks about a concrete problem like, "How does code complete work?"
A good white board interview question asks about a concrete problem like, "How does code complete work?"
You don't know that they're judging you based on perfect syntax and probably they aren't. Just because someone points out a syntax error doesn't mean they're marking you down for it; maybe they're trying to be helpful.
But that's the problem; there's little or no feedback from most interviews (for legal and other reasons), so many candidates assume the worst.
Yes, the key is to turn it into a technical conversation. Pretend they don't know the answer and genuinely want your help. Interviews are artificial but talking about technical problems in front of whiteboards is a normal work activity. It's not about perfect code but more about coming across as someone they'd want to go to for technical advice on a tough problem.
But that's the problem; there's little or no feedback from most interviews (for legal and other reasons), so many candidates assume the worst.
Yes, the key is to turn it into a technical conversation. Pretend they don't know the answer and genuinely want your help. Interviews are artificial but talking about technical problems in front of whiteboards is a normal work activity. It's not about perfect code but more about coming across as someone they'd want to go to for technical advice on a tough problem.