The Technical Interview Is Dead(techcrunch.com)
techcrunch.com
The Technical Interview Is Dead
http://techcrunch.com/2013/06/22/the-technical-interview-is-dead/
131 comments
I would agree with this in that the "audition project" would not be done by top candidates - but I'd like to suggest that they already do the audition project.
The group I'm in (all flash SAN storage) has gone from single digits to ~80 people in under a year, and basically the only predictive metric is referrals from people who have made other good referrals. We're writing a complete HPC OS, and it's sink or swim. The referrals are almost all swimming.
That's it. Almost all other hires have been completely random in quality.
So these amazing engineers that you're really struggling to hire are actually being hired because of a career long audition with their peers. It's wildly incestuous, and has some interesting effects. Some effects are bad: no one knows new tools/new development styles. Some effects are good: at the top, gender split is much closer to 50/50.
I'd love to figure out how to vet someone in under 3 months that is not part of this professional tribe, but even typing that out I'm almost suspicious they'd succeed and become part of it if they aren't in the tribe already. It seems like even my attitudes are a problem in opening up these kinds of candidate searches.
The group I'm in (all flash SAN storage) has gone from single digits to ~80 people in under a year, and basically the only predictive metric is referrals from people who have made other good referrals. We're writing a complete HPC OS, and it's sink or swim. The referrals are almost all swimming.
That's it. Almost all other hires have been completely random in quality.
So these amazing engineers that you're really struggling to hire are actually being hired because of a career long audition with their peers. It's wildly incestuous, and has some interesting effects. Some effects are bad: no one knows new tools/new development styles. Some effects are good: at the top, gender split is much closer to 50/50.
I'd love to figure out how to vet someone in under 3 months that is not part of this professional tribe, but even typing that out I'm almost suspicious they'd succeed and become part of it if they aren't in the tribe already. It seems like even my attitudes are a problem in opening up these kinds of candidate searches.
You make an interesting macro point. But there's a world of difference between an entry level full-time job and a 1-2 week "audition project".
The phenomenon you're describing is what makes recruiting such an interesting problem. The largest or wealthiest employers can often afford to constrain their candidates to people who have proven themselves in other settings. Which of course leaves a "Moneyball"-type opening to the employers who can figure out how to hire for aptitude instead of experience. Which makes hiring at those savvier employees a high-stakes, high-value process, instead of a tedious formality inflicted across the whole team.
The phenomenon you're describing is what makes recruiting such an interesting problem. The largest or wealthiest employers can often afford to constrain their candidates to people who have proven themselves in other settings. Which of course leaves a "Moneyball"-type opening to the employers who can figure out how to hire for aptitude instead of experience. Which makes hiring at those savvier employees a high-stakes, high-value process, instead of a tedious formality inflicted across the whole team.
> Which of course leaves a "Moneyball"-type opening to the employers who can figure out how to hire for aptitude instead of experience.
I want to call for tokenadult's canned "within the US, look for a work sample. Outside the US, use an IQ test" post. We figured out _how_ to hire for aptitude a long time ago (subject, of course, to the fact that prediction is never perfect); Malcom McLean apparently filled his new shipping company top-down by who, among the people who worked for his old trucking company, scored highest on an IQ test. But then we decided that that would be illegal in the future?
I want to call for tokenadult's canned "within the US, look for a work sample. Outside the US, use an IQ test" post. We figured out _how_ to hire for aptitude a long time ago (subject, of course, to the fact that prediction is never perfect); Malcom McLean apparently filled his new shipping company top-down by who, among the people who worked for his old trucking company, scored highest on an IQ test. But then we decided that that would be illegal in the future?
We know what the solution looks like, but not what its precise measurements are; the person who comes up with a universal, economical, predictively powerful work sample test for software developers is probably going to make a lot of money.
Really a family of such metrics because the moment the first one hits the ground in any earnestly it'll be gamed to death. Nothing ruins a powerful, economical, repeatable indicator for predicting economic activity faster than using it.
The point I was trying to make wasn't about hiring "for experience" but leveraging those who were around them to vouch for their aptitude. This is a distinct difference, and you have to work hard to get people to think in these terms instead of amount of experience. I may be naive in even thinking we're accomplishing this.
Without the public displays of performance that baseball has, there's another level of indirection that you must filter through to try and illustrate a candidates performance.
This makes it something like the "Moneyball of Moneyball of programming".
Without the public displays of performance that baseball has, there's another level of indirection that you must filter through to try and illustrate a candidates performance.
This makes it something like the "Moneyball of Moneyball of programming".
Even worse is the unpaid audition project. I've had a prospective employer try to pitch me on that and they were shocked when I balked.
Technical interviews are perfectly good at doing what they are intended to do: screen out people who can't program. Somewhere between 50% and 80% of people I meet in first-round interviews literally just cannot program. Half of the people who say C++ is their strongest language can't tell me the difference between a pointer and a reference, or between a pointer to const and a const pointer.
I don't know if this problem is specific to our industry or not. Do people apply to be English teachers when it takes them an hour and a stack of reference material to form one English sentence? Do people apply to be carpenters who don't know how to hold a saw?
I don't know if this problem is specific to our industry or not. Do people apply to be English teachers when it takes them an hour and a stack of reference material to form one English sentence? Do people apply to be carpenters who don't know how to hold a saw?
Very good point about screening.
When I was interviewing people, phone screen with some basic concept questions would filter out 80% of applicants with nicely looking resumes; an hour-long whiteboard screen with FizzBuzz kind of problems would take care of the rest. Very quick and efficient process.
When I was interviewing people, phone screen with some basic concept questions would filter out 80% of applicants with nicely looking resumes; an hour-long whiteboard screen with FizzBuzz kind of problems would take care of the rest. Very quick and efficient process.
If the end result - filter out every applicant -- I have quicker and more efficient process in mind.
I'm not a C++ programmer; what is the difference between a const pointer and a reference?
Pointer allows you to dereference a memory address. A reference is the physical memory address variable.
If you have a variable and you want to pass-by-reference, making any changes to the original variable effect the originally allocated value.
You use a pointer for instance if you want to dereference a physical memory address as part of an array of allocated memory addresses.
If you have a variable and you want to pass-by-reference, making any changes to the original variable effect the originally allocated value.
You use a pointer for instance if you want to dereference a physical memory address as part of an array of allocated memory addresses.
// reference example
int value = 0;
int value1 = &value;
value1 = 2;
// value should equal 2
// pointer example
int value[5] = {1,2,3,4,5};
*value += 1;value++;
*value += 1;value++;
*value += 1;value++;
*value += 1;value++;
*value += 1;
// should now be {2,3,4,5,6}I'm not c++ programmer, so please correct me if I'm wrong, but I think that the reference example is not correct.
Shouldn't it be as bellow?
// reference example
int value = 0;
int& value1 = value;
value1 = 2;
// value should equal 2Yes. And this subthread nicely proves the parent: At least half of people who claim to program in C++ can't.
That is more correct, yes.
The syntax is different, but in terms of what you can do, the only difference I know is that references can be reassigned.
Both give you the guarantee that it's not pointing at the NULL address. In the case of const pointers, that's because you can't reassign it, and you'd never deliberately assign a const pointer to NULL. In the case of references, it's because there's actually no such thing as a null reference in C++.
That's not the case in Java and C#, which allow "null" to be a right-hand value for all types.
Both give you the guarantee that it's not pointing at the NULL address. In the case of const pointers, that's because you can't reassign it, and you'd never deliberately assign a const pointer to NULL. In the case of references, it's because there's actually no such thing as a null reference in C++.
That's not the case in Java and C#, which allow "null" to be a right-hand value for all types.
>the only difference I know is that references can be reassigned.
Uhh no. Neither the const pointer nor the reference can be reassigned. And you have it the wrong way around. References cannot be null, pointers can.
The other difference is that if you try and dynamic_cast a reference and it failed, it throws an exception where as a pointer just returns nullptr.
>That's not the case in Java and C#, which allow "null" to be a right-hand value for all types.
That's also wrong, primitives are not nullable.
Uhh no. Neither the const pointer nor the reference can be reassigned. And you have it the wrong way around. References cannot be null, pointers can.
The other difference is that if you try and dynamic_cast a reference and it failed, it throws an exception where as a pointer just returns nullptr.
>That's not the case in Java and C#, which allow "null" to be a right-hand value for all types.
That's also wrong, primitives are not nullable.
Sorry, somehow I managed to forget that primitives existed for that brief moment. I was reacting to the realization that "null" is a bizarre special case for the type system.
I was wrong on reinitializing references as well. Thanks for pointing that out. As far as const pointers, I meant that they were unlikely to be null, since they would have to be initialized to null. Of course, then I realized that you could easily assign them to another null pointer that you didn't realize was null.
Not a good day for me, but I don't think it warrants that kind of snarkiness.
To my credit, please reread the part where I said "there is actually no such thing as a null reference."
I was wrong on reinitializing references as well. Thanks for pointing that out. As far as const pointers, I meant that they were unlikely to be null, since they would have to be initialized to null. Of course, then I realized that you could easily assign them to another null pointer that you didn't realize was null.
Not a good day for me, but I don't think it warrants that kind of snarkiness.
To my credit, please reread the part where I said "there is actually no such thing as a null reference."
Well there is Nullable<T> for C#. But I haven't touched that language in ages.
The best part of your reply is in showing how he failed his own interview, figuratively speaking of course. In real life, the candidate doesn't get to explain things the day after they were turned down because of bad answers.
I truly believe he knows his stuff, but even he made a mistake. He, however, didn't have to come up with his answer while people are staring at him.
I truly believe he knows his stuff, but even he made a mistake. He, however, didn't have to come up with his answer while people are staring at him.
[deleted](1)
A reference looks like a value but is actually a point (de-references are automatic). Most high-level languages prefer references over pointers as they are harder to screw up. E.g.
myref = otherref + 10
is equivalent to
* myptr = * otherptr + 10
Now, most languages also have a notion of values that are neither pointers nor references, and primitives generally are expressed as values, but hopefully you get the point.
myref = otherref + 10
is equivalent to
* myptr = * otherptr + 10
Now, most languages also have a notion of values that are neither pointers nor references, and primitives generally are expressed as values, but hopefully you get the point.
[deleted]
A const pointer could be NULL, among many other differences.
I wish I interviewed where you do. It is more like 90% for me.
Agree pretty much completely with this.
I don't think I saw this mentioned but another technique that companies use it to subject you to formal tests. Usually produced by some 3rd party. I have huge experience with Excel and spreadsheets since Lotus 123 and have worked on and produced some enormously complicated, but successful spreadsheet/VBA solutions. I am almost invariably seen as the resident Excel expert in all the organizations I have been employed, yet have failed a test to prove my Excel "competence" because somehow the Dictionary Object was considered all important by this 3rd party test and yet in 15 years of spreadsheet is never something I touched.
I don't doubt that the Dictionary object is very useful in some use-cases but to fail an Excel competence test because some test producer decided to get cute is hardly an indication of real world knowledge and experience with a product.
That is just Excel, I shudder to think what a test in C++ would do to even a highly experienced C++ programmer if it decided to explore all the edge cases and obscure gotcha's that exist in C++.
This is not to say that some tests cannot be useful - just be very careful of what you are actually testing for and how you are doing it.
I don't think I saw this mentioned but another technique that companies use it to subject you to formal tests. Usually produced by some 3rd party. I have huge experience with Excel and spreadsheets since Lotus 123 and have worked on and produced some enormously complicated, but successful spreadsheet/VBA solutions. I am almost invariably seen as the resident Excel expert in all the organizations I have been employed, yet have failed a test to prove my Excel "competence" because somehow the Dictionary Object was considered all important by this 3rd party test and yet in 15 years of spreadsheet is never something I touched.
I don't doubt that the Dictionary object is very useful in some use-cases but to fail an Excel competence test because some test producer decided to get cute is hardly an indication of real world knowledge and experience with a product.
That is just Excel, I shudder to think what a test in C++ would do to even a highly experienced C++ programmer if it decided to explore all the edge cases and obscure gotcha's that exist in C++.
This is not to say that some tests cannot be useful - just be very careful of what you are actually testing for and how you are doing it.
Often enough, just testing the things that everyone uses day in and day out is plenty enough to tell you about how a person can be as a programmer. Such as creating a table view that says 1 to 5 or animating things in iOS, and const pointers and references in C++
I've mentioned this before, but I have a trendy skill, and one that has a higher barrier to entry than eg web developer. If a potential employer says, "so... x0x0. What we're thinking is that you burn a week of vacation time doing a trial project for us" my response is something like "hahahaha...bzzt. next"
But but but we'll pay you! We'll pay you approximately half the market rate for your services in a consulting setting! How could you turn that down?
What's your skill? Machine learning? Big Data?
ml with a bunch of hadoop knowledge
As mentioned in the article, after Google examined the data, they found that technical interviews were worthless at determining employee quality. Instead, they say, "what works well are structured behavioral interviews." This seems pretty damning for the technical interview. Would you just say that Google was doing technical interviews wrong?
It's funny - when I hear people in the press discussing technical interviews [1] I hear questions like this: "How many golf balls could you fit into a school bus?"
When I attend technical interviews I hear questions like this: "How would you implement an arbitrary-precision decimal number class, like Java's BigDecimal? Can you write out the add function on that whiteboard?"
When Google's Laszlo Bock [2] says "On the hiring side, we found that brainteasers are a complete waste of time. How many golf balls can you fit into an airplane? How many gas stations in Manhattan? A complete waste of time. They don’t predict anything." it seems fairly clear to me the idea he's dismissing is brainteasers, not whiteboard coding.
That's consistent with other reports [3,4] from a few years ago saying brain teasers were not used.
The author of the TechCrunch article seems to interpret Bock as saying whiteboard coding is an ineffective practice. I'm not convinced the Bock interview said that.
[1] http://online.wsj.com/article/SB1000142405297020455230457711... [2] https://www.nytimes.com/2013/06/20/business/in-head-hunting-... [3] http://www.technologywoman.com/2010/05/17/debunking-the-goog... [4] http://www.guardian.co.uk/technology/us-news-blog/2012/oct/0...
When I attend technical interviews I hear questions like this: "How would you implement an arbitrary-precision decimal number class, like Java's BigDecimal? Can you write out the add function on that whiteboard?"
When Google's Laszlo Bock [2] says "On the hiring side, we found that brainteasers are a complete waste of time. How many golf balls can you fit into an airplane? How many gas stations in Manhattan? A complete waste of time. They don’t predict anything." it seems fairly clear to me the idea he's dismissing is brainteasers, not whiteboard coding.
That's consistent with other reports [3,4] from a few years ago saying brain teasers were not used.
The author of the TechCrunch article seems to interpret Bock as saying whiteboard coding is an ineffective practice. I'm not convinced the Bock interview said that.
[1] http://online.wsj.com/article/SB1000142405297020455230457711... [2] https://www.nytimes.com/2013/06/20/business/in-head-hunting-... [3] http://www.technologywoman.com/2010/05/17/debunking-the-goog... [4] http://www.guardian.co.uk/technology/us-news-blog/2012/oct/0...
I'm suspicious of whiteboard coding as well. Why not give the person a computer and 20 minutes? Why make it less realistic by taking all reference material, editor and ability to test-run the program?
This
It's the same stupid behaviour with "scrum post-its" when there are tools available.
Yes, you can't interview a surgeon by giving him a scalpel and a patient, but this is not the case with software development interviews
It's the same stupid behaviour with "scrum post-its" when there are tools available.
Yes, you can't interview a surgeon by giving him a scalpel and a patient, but this is not the case with software development interviews
No, what Google said was that brain-teasers were useless. Which is of course something many people have been saying for years.
No, I read tptacek as agreeing with this person at Google. [1] That person is trying to develop a better interview formalism that works and is repeatable. So is tptacek, albeit probably in a slightly different direction.
What neither the Googlers nor Ptacek are saying is that the solution to the screening problem is to ask candidates to accept a deal that most full-time employees, and most clueful contractors too, would reject out of hand.
If your screening strategy would reject any sizable percentage of people who are currently successfully working for someone else, your screening strategy is bogus. Unless your express goal is to hire promising juniors who don't know their own value. Which does feel like the goal of many hiring managers: As market rates get bid up, they're desperately looking for people who are just clueful enough to succeed, yet just clueless enough not to ask for money in return.
----
[1] I do not presume that any single person speaks for all of Google. In particular, I would be surprised if that giant company of folks who were all hired via brainteaser interviews has actually entirely stopped using brainteaser interviews. Fixing interviewing culture is hard.
What neither the Googlers nor Ptacek are saying is that the solution to the screening problem is to ask candidates to accept a deal that most full-time employees, and most clueful contractors too, would reject out of hand.
If your screening strategy would reject any sizable percentage of people who are currently successfully working for someone else, your screening strategy is bogus. Unless your express goal is to hire promising juniors who don't know their own value. Which does feel like the goal of many hiring managers: As market rates get bid up, they're desperately looking for people who are just clueful enough to succeed, yet just clueless enough not to ask for money in return.
----
[1] I do not presume that any single person speaks for all of Google. In particular, I would be surprised if that giant company of folks who were all hired via brainteaser interviews has actually entirely stopped using brainteaser interviews. Fixing interviewing culture is hard.
Has Google abandoned whiteboard coding interviews for developers? Unless the answer is yes, I think you might be misreading what Bock said.
Google has not abandoned whiteboard coding interviews for developers. From within the last year:
http://www.google.com/about/jobs/lifeatgoogle/hangout-on-air...
http://www.google.com/about/jobs/lifeatgoogle/hangout-on-air...
If I'm a top coder I'd do the 3 months evaluation contract in a heartbeat... why? Well because if I'm a top coder then chances are I know I'm pretty good, and by the end of three months I'll be able to better assess the value I can (maybe already do) bring to the company. That would in turn give me a ton more leverage for compensation negotiations than I would have had earlier. Also, if I'm hoping to find a place to park for a while, isn't it better for both sides to get a better feel for each other by actually experiencing that fit?
If it doesn't work out there's always the next 3 month interview...
If it doesn't work out there's always the next 3 month interview...
It's getting harder to find good developers, not easier
This could be part of what's going on. Small companies are getting the leavings of the big ultra-competitive companies. Small companies notice no matter who they hire, they are never up to par. Small companies blame this on the technical interview.
This could be part of what's going on. Small companies are getting the leavings of the big ultra-competitive companies. Small companies notice no matter who they hire, they are never up to par. Small companies blame this on the technical interview.
Another compounding effect is that in order to get good at hiring, you need years of experience hiring, so you can find out what kind of interview is predictive and what kind of interview just wastes time. Most small companies don't have that kind of history to draw on.
Also, when you are able to detect and hire the best people you can put a heavy pressure on the competition by simultaneously depriving them of talent and attracting more talent yourself. E.g. if the company A that has 8 out of 10 top people in my field made me an offer then the B with 0 such people will have to make a much better offer for me to even consider them.
In such a situation the usual "false negative is okay, it cost more to hire a wrong person than to skip a right one" stops working. Skipping a right one will cost a lot in the long run.
In such a situation the usual "false negative is okay, it cost more to hire a wrong person than to skip a right one" stops working. Skipping a right one will cost a lot in the long run.
I would say that a technical interview consisting of coding questions that test data structures and algorithms is about as standard as you can get. True that some interviewers will turn this into a hazing ritual, but hopefully there's some feedback mechanism by which that behavior can be discouraged -- a few ideas would be to have shadow interviewers or allowing candidates to submit feedback about the interview content and process etc.
IMHO, the ideal technical interview would be one where the interviewer is able to dig in deep into the candidate's experience and able to ask intelligent questions that reveal the candidate's technical depth. But this requires very strong interviewing skills as well as an appropriate technical background on part of the interviewer -- this will not scale.
The "audition project" is a non-starter for the reasons you mentioned and as you correctly point out, we have no evidence that it works in the large.
So back we are to the flawed coding interview. Like some wag said about democracy: it's a shitty system, but there's nothing better.
IMHO, the ideal technical interview would be one where the interviewer is able to dig in deep into the candidate's experience and able to ask intelligent questions that reveal the candidate's technical depth. But this requires very strong interviewing skills as well as an appropriate technical background on part of the interviewer -- this will not scale.
The "audition project" is a non-starter for the reasons you mentioned and as you correctly point out, we have no evidence that it works in the large.
So back we are to the flawed coding interview. Like some wag said about democracy: it's a shitty system, but there's nothing better.
I didn't say "standard", I said "standardized".
A standardized interview process is one where every candidate receives (roughly) the same set of questions, so that the company can look over the last 18 months of interviews, compare every candidate regardless of who gave the interview, and make observations across candidates.
A rigorous interview (for lack of a better term) is one where questions have either correct answers or quantifiable answers. "Best algorithmic approach" to a problem is usually not a rigorous question (unless the problem domain is very well defined and very well studied). But "number of valid algorithmic approaches offered" is, as would be "validated time complexity of best offered approach".
The idea that a tech interview should be digging deep into a candidate's background to assess their experience and derive intelligent questions that reveal depth is, I believe, the pervasive myth underlying all bad technical hiring. The people who conduct technical interviews are developers, not trained psychologists. Developers have a pronounced tendency to overestimate their own confidence in things like interviews. For lack of a more eloquent way to put this: developers suck at interviewing.
There's no better way I know of to interview devs than to have devs do interviews, because it takes a trained developer brain to accurately classify answers to useful tech questions (ie, "is this unexpected response incorrect, or is it a correct response that uses a more clever approach than the one we expected, or is it a correct response phrased in different terminology than we're used to", &c). You have to have devs doing interviews.
So the next best thing is to stop pretending that devs are capable of doing something that they're mostly not capable of doing. To me, that says "take a couple weeks to come up with a standardized interview that every developer will give".† Watch the responses to those interviews. Look at who those interviews select; if those people seem to be working out, stick with it, else adjust.
† One indicator I'm playing with: if anyone on your team looks forward to doing interviews, you're doing something wrong; for the interviewer, the process should be mostly very boring.
A standardized interview process is one where every candidate receives (roughly) the same set of questions, so that the company can look over the last 18 months of interviews, compare every candidate regardless of who gave the interview, and make observations across candidates.
A rigorous interview (for lack of a better term) is one where questions have either correct answers or quantifiable answers. "Best algorithmic approach" to a problem is usually not a rigorous question (unless the problem domain is very well defined and very well studied). But "number of valid algorithmic approaches offered" is, as would be "validated time complexity of best offered approach".
The idea that a tech interview should be digging deep into a candidate's background to assess their experience and derive intelligent questions that reveal depth is, I believe, the pervasive myth underlying all bad technical hiring. The people who conduct technical interviews are developers, not trained psychologists. Developers have a pronounced tendency to overestimate their own confidence in things like interviews. For lack of a more eloquent way to put this: developers suck at interviewing.
There's no better way I know of to interview devs than to have devs do interviews, because it takes a trained developer brain to accurately classify answers to useful tech questions (ie, "is this unexpected response incorrect, or is it a correct response that uses a more clever approach than the one we expected, or is it a correct response phrased in different terminology than we're used to", &c). You have to have devs doing interviews.
So the next best thing is to stop pretending that devs are capable of doing something that they're mostly not capable of doing. To me, that says "take a couple weeks to come up with a standardized interview that every developer will give".† Watch the responses to those interviews. Look at who those interviews select; if those people seem to be working out, stick with it, else adjust.
† One indicator I'm playing with: if anyone on your team looks forward to doing interviews, you're doing something wrong; for the interviewer, the process should be mostly very boring.
> In this environment, why would a top developer, who has their choice among tens of different high-paying interesting jobs, moonlight for a prospective new employer just so they can make sure the relationship's going to work?
I have my doubts that I'm a top developer (I have a long list of things I'd like to understand/do better in order to consider myself in that category), but I've done and would consider doing a paid audition project.
The job I have right now pays market rates, offers good benefits, and involves work for famous clients. It's not a bad gig, and there are enough people who've been there for over five years that it's clearly a good environment for some. On the other hand, I don't find the work itself particularly satisfying. Most of our projects themselves are a bit staid (as projects from established brand clients who feel they have more to lose than gain from risky moves can be), and a lot of what I've ended up doing is production front-end work rather than much problem solving.
This is different from the picture I got during the interviews. That could be as much about my own ability to read what day-to-day life was likely to be like post-hire as anything else, but whether it's uniquely difficult for me or for everybody, I'm left with the same problem -- how do I get a glimpse of what working there is really going to be like?
Seems to me a paid audition is one good answer.
I'd rather do 2-3 months on contract than a 2-3 day project, but I'd rather do the 2-3 days of real work than nothing.
And I think the 2-3 days of real work idea beats another practice that's becoming prevalent: the ~1 day (unpaid!) work sample/code challenge. I don't mind such challenges in and of themselves; a small exercise can be educational and an opportunity to use all the things you've been thinking would be good ideas to fold into a new project. What I dislike is that real feedback seems to be scarce. Even the people who've hired me didn't give me much feedback beyond "it met the requirements and we learned a thing or two from your work." By the time I've invested a workday of focused time in a company, I'd appreciate a "we were hoping to see" or "here's what we liked", and I'm finding I don't get much of that from challenges/exercises. Whatever the flaws of the conventional interview process or the paid real project may be, at least they're both generally more symmetric in terms of investments required and opportunities for evaluation.
I have my doubts that I'm a top developer (I have a long list of things I'd like to understand/do better in order to consider myself in that category), but I've done and would consider doing a paid audition project.
The job I have right now pays market rates, offers good benefits, and involves work for famous clients. It's not a bad gig, and there are enough people who've been there for over five years that it's clearly a good environment for some. On the other hand, I don't find the work itself particularly satisfying. Most of our projects themselves are a bit staid (as projects from established brand clients who feel they have more to lose than gain from risky moves can be), and a lot of what I've ended up doing is production front-end work rather than much problem solving.
This is different from the picture I got during the interviews. That could be as much about my own ability to read what day-to-day life was likely to be like post-hire as anything else, but whether it's uniquely difficult for me or for everybody, I'm left with the same problem -- how do I get a glimpse of what working there is really going to be like?
Seems to me a paid audition is one good answer.
I'd rather do 2-3 months on contract than a 2-3 day project, but I'd rather do the 2-3 days of real work than nothing.
And I think the 2-3 days of real work idea beats another practice that's becoming prevalent: the ~1 day (unpaid!) work sample/code challenge. I don't mind such challenges in and of themselves; a small exercise can be educational and an opportunity to use all the things you've been thinking would be good ideas to fold into a new project. What I dislike is that real feedback seems to be scarce. Even the people who've hired me didn't give me much feedback beyond "it met the requirements and we learned a thing or two from your work." By the time I've invested a workday of focused time in a company, I'd appreciate a "we were hoping to see" or "here's what we liked", and I'm finding I don't get much of that from challenges/exercises. Whatever the flaws of the conventional interview process or the paid real project may be, at least they're both generally more symmetric in terms of investments required and opportunities for evaluation.
I'm not saying that freelancing is bad. For a bunch of reasons, I'm very positive about freelancing and consulting.
You sound like someone who'd rather be consulting than working full-time jobs. Great! You'll probably also make more money and get more professional experience faster than people who take the less-risky full-time job path.
What I object to is the notion of temporarily converting full-time workers into consultants so that companies can mitigate their concerns about how inept their own technical interviews are. Consultants and full-timers are two very different things, and they aren't easily convertible.
You sound like someone who'd rather be consulting than working full-time jobs. Great! You'll probably also make more money and get more professional experience faster than people who take the less-risky full-time job path.
What I object to is the notion of temporarily converting full-time workers into consultants so that companies can mitigate their concerns about how inept their own technical interviews are. Consultants and full-timers are two very different things, and they aren't easily convertible.
> the ~1 day (unpaid!) work sample/code challenge [...] Even the people who've hired me didn't give me much feedback beyond "it met the requirements and we learned a thing or two from your work."
You need to be concerned about the employers who "learn a thing or two from your work" and don't hire you.
Some programming "team leaders" I've met wouldn't hesitate to bring in applicants to work unpaid for a day on coding challenges containing technical blockers from their own projects.
And of course much of what happens in the open source programming world is the same on a larger scale without even needing to supply office space for a day.
You need to be concerned about the employers who "learn a thing or two from your work" and don't hire you.
Some programming "team leaders" I've met wouldn't hesitate to bring in applicants to work unpaid for a day on coding challenges containing technical blockers from their own projects.
And of course much of what happens in the open source programming world is the same on a larger scale without even needing to supply office space for a day.
I have my doubts that I'm a top developer (I have a long list of things I'd like to understand/do better in order to consider myself in that category), but I've done and would consider doing a paid audition project.
The problem is that it's not always your choice to make. Noncompete clauses exist, and, if you're outside the state of California, are generally enforceable. I don't know about you, but I'd really rather have a system where taking an interview wasn't a fire-able offence.
The problem is that it's not always your choice to make. Noncompete clauses exist, and, if you're outside the state of California, are generally enforceable. I don't know about you, but I'd really rather have a system where taking an interview wasn't a fire-able offence.
To spot a top performer one needs no elaborate testing or, god forbid, standardized interviews. It is the same as it is in sports or music or arts - there are cues that identify a top performer. What standardized tests you want to apply to, say, Beethoven, Orwell or Muhammad Ali, or, good forbid, Steve Jobs, or Igor Sysoev?)
It is not about testing, it is about having a gift (or not) and a character - willingness to become a master just for the sake of ones love/passion/enjoyment of the craft, not any stupid job or position.
It is not about testing, it is about having a gift (or not) and a character - willingness to become a master just for the sake of ones love/passion/enjoyment of the craft, not any stupid job or position.
[deleted]
[deleted]
As a shameless plug - if anyone's interested in a completely in-browser experience where you and a candidate can both write and EXECUTE code simultaneously, I make a tool for that: https://coderpad.io
Why do I want to watch my candidate write code? Doesn't that just make them incredibly tense? Isn't that queering my results?
> these teams are hired by technical interviews.
The "audition project" is a trend story with, from what I can see, very little empirical evidence to back it up.
The problem with the "audition project" is precisely that it has the same inconvenient economics as dating, finding an apartment, or job hunting/hiring in general, only more so. There is greatly increased "opportunity cost." This alone can explain the lack of data. It might be like a lot of "better" tests -- it might be a better way to gauge suitability on an instance-ti-instance basis, but the economics preclude it from being widely used.
The problem with the "audition project" is precisely that it has the same inconvenient economics as dating, finding an apartment, or job hunting/hiring in general, only more so. There is greatly increased "opportunity cost." This alone can explain the lack of data. It might be like a lot of "better" tests -- it might be a better way to gauge suitability on an instance-ti-instance basis, but the economics preclude it from being widely used.
Since you are an employer and interviewer yourself, have you thought about writing a guide to doing technical interviews? Or do you think one of the existing guides, such as SPolsky's Guerrilla Guide, is good enough?
The approach tptacek and Matasano take is explained (for a potential candidate's benefit) on the Matasano site: http://www.matasano.com/careers/
I think that's kind of the point. Employer or not, no one really has the answer to effective interviews.
Also, if "audition projects" were to become the norm then you'd see rampant cheating. Simply farm out the project to a competent freelancer and then pass it off as your own. The freelancer may not even know they're working on an audition project. And they may not even end up being paid more than the candidate since they could likely finish the project in much less time (and thus work at a much higher effective hourly rate).
And how do you filter the candidates who you give audition projects to in the first place? That's very difficult without a technical interview.
And how do you filter the candidates who you give audition projects to in the first place? That's very difficult without a technical interview.
Detecting a cheat should be easy -- just ask them to walk through the project and explain it. You should be able to expose a cheater with a couple of deep questions.
But then wouldn't that be a "technical interview" and therefore bad?
Sigh, this is what I was going to say (I tend to say it with more typos :-).
The interesting effects here for people building teams then are in the dispossessed. The folks who didn't graduate college (so shunned by many of the big names) or the folks who took an off beaten path. My concern is that if Google fixes their blind spot with respect to this talent pool it will get that much harder for startups. So far I've not seen any indication that they have but I worry about it.
[Edit: I can't even joke without getting it wrong.]
The interesting effects here for people building teams then are in the dispossessed. The folks who didn't graduate college (so shunned by many of the big names) or the folks who took an off beaten path. My concern is that if Google fixes their blind spot with respect to this talent pool it will get that much harder for startups. So far I've not seen any indication that they have but I worry about it.
[Edit: I can't even joke without getting it wrong.]
If you've found a superstar, then surely they can skip the process. Otherwise a test drive sounds perfect. It's what you'd do in any other transaction.
Yeah? Test drive a lot of houses, do you?
The reason you get to test drive cars is that the people who have cars to sell are desperate to sell them to you.
The reason you get to test drive cars is that the people who have cars to sell are desperate to sell them to you.
That's one reason. However, I have been in the position of being desperate to sell a home (many people were at the time and in some places still are, I'm sure), and "test drive" or any equivalent simply does not make sense in some situations.
There's a lot less overhead to driving a car for ten minutes than there is to living in a new place for a day or two... to get the right impression of a car, you drive it in a residential area, you drive it on the highway, and there, you have at least an overview. To do the same for a house? Move in, settle in, meet the neighbors, live in it, then move out, because someone else wants a test drive, too.
And, of course, neither applies well at all to interviewing a person.
The metaphors just don't work here, and they're not helpful. Maybe there are things that work while car/house/pet/food shopping that can be applied to interviewing, but that correlation does not necessarily indicate any similarity between those situations, and comparison isn't useful unless the activities themselves actually are meaningfully comparable.
There's a lot less overhead to driving a car for ten minutes than there is to living in a new place for a day or two... to get the right impression of a car, you drive it in a residential area, you drive it on the highway, and there, you have at least an overview. To do the same for a house? Move in, settle in, meet the neighbors, live in it, then move out, because someone else wants a test drive, too.
And, of course, neither applies well at all to interviewing a person.
The metaphors just don't work here, and they're not helpful. Maybe there are things that work while car/house/pet/food shopping that can be applied to interviewing, but that correlation does not necessarily indicate any similarity between those situations, and comparison isn't useful unless the activities themselves actually are meaningfully comparable.
Most startups aren't looking to hire developers that are desperate for jobs. The point is they have a job they like reasonably well, they have other offers on the table from other companies. Other companies here did not require them to moonlight for a week knowing full well it might not work out (keep in mind they would be moon lighting with you in addition to interviewing with other companies). So why would they agree to moonlight if -- even without moonlighting -- the job offer is already a tough sell?
There are two possibilities:
1) You're looking for a co-founder of for perhaps the founding engineer, e.g., Sergey Brin and Larry Page hiring Craig Silverstein. You've impressed them to a great degree (they like your product demo, your background), and they are already willing to pay potential opportunity costs (a job at an established company, finishing their Ph.D. program, etc...) to join you. Then this really isn't a hiring problem per-se, this is a "finding a co-founder or founding engineer" problem; you are less likely be hiring someone "from the cold" for this in the first place (this is why so many companies are started by graduate students or former coworkers...)
2) You're willing to hire people who are unable to find other full-time work, in other words someone with no formal experience or experience in a completely wrong area (e.g., someone who worked on internal business web applications, but who wants to get into embedded programming).
Then you're really looking to hire an intern (in case of no formal experience) -- where the process is essentially that, a contract to hire, or simply take a calculated risk. In other words, you are able to see potential in this person (which they've demonstrated by perhaps building an app in their spare time) and are willing to set them up for success (as oppose to trying to find the best candidates out of a pool).
There are two possibilities:
1) You're looking for a co-founder of for perhaps the founding engineer, e.g., Sergey Brin and Larry Page hiring Craig Silverstein. You've impressed them to a great degree (they like your product demo, your background), and they are already willing to pay potential opportunity costs (a job at an established company, finishing their Ph.D. program, etc...) to join you. Then this really isn't a hiring problem per-se, this is a "finding a co-founder or founding engineer" problem; you are less likely be hiring someone "from the cold" for this in the first place (this is why so many companies are started by graduate students or former coworkers...)
2) You're willing to hire people who are unable to find other full-time work, in other words someone with no formal experience or experience in a completely wrong area (e.g., someone who worked on internal business web applications, but who wants to get into embedded programming).
Then you're really looking to hire an intern (in case of no formal experience) -- where the process is essentially that, a contract to hire, or simply take a calculated risk. In other words, you are able to see potential in this person (which they've demonstrated by perhaps building an app in their spare time) and are willing to set them up for success (as oppose to trying to find the best candidates out of a pool).
Yes, you can test drive the neighborhood at least--in the sublet section of craigslist.
Variations on the theme of "have them do a paid project for you" surface here every so often. The problem is that a large number of job seekers currently have a job (everyone keeps telling them it is easier to get a job while you have a job). Even qualified, competent programmers will find it hard to accept the risk of being dropped at the conclusion of the short project (and if you no-one ever gets dropped, what is the point of the project besides giving you a warm and fuzzy?)
Fact is, you have to do everything up to that 'mini project' then just go with your gut and hire/no hire. If they underperform in the role you need to either change the role, try and train them up or let them go.
This trial period is just a way of pushing the risk of a bad hiring decision back onto the candidate and allowing the manager to avoid the discomfort of actually firing someone (although telling the candidate that the project didn't work out is functionally the same thing).
Fact is, you have to do everything up to that 'mini project' then just go with your gut and hire/no hire. If they underperform in the role you need to either change the role, try and train them up or let them go.
This trial period is just a way of pushing the risk of a bad hiring decision back onto the candidate and allowing the manager to avoid the discomfort of actually firing someone (although telling the candidate that the project didn't work out is functionally the same thing).
Yes, I think this is the main problem with the idea: it doesn't make sense for someone who already has a job to do a trial job.
That's why I think you already do see it with one large category of tech job-seekers where that isn't a problem: students. Tech companies hire student interns for a summer effectively as a trial period, and both sides often like that arrangement.
That's why I think you already do see it with one large category of tech job-seekers where that isn't a problem: students. Tech companies hire student interns for a summer effectively as a trial period, and both sides often like that arrangement.
Yeah, I think it's a nice solution but it only works in very narrow circumstances.
Once I was given a much smaller, unpaid task as part of the interview process. It wasn't a puzzle or trick, but something actually useful. All told it took me a couple of hours one evening. You might be thinking, "How much can you learn about someone from such a small task?" It depends on the task, of course. Mine was typical network code, with a couple of non-obvious but absolutely real-world edge cases. I spent time making sure my comments were great, and then wrote a man page. I think they learned some important things about me, and on my end it was an easily paid investment.
Once I was given a much smaller, unpaid task as part of the interview process. It wasn't a puzzle or trick, but something actually useful. All told it took me a couple of hours one evening. You might be thinking, "How much can you learn about someone from such a small task?" It depends on the task, of course. Mine was typical network code, with a couple of non-obvious but absolutely real-world edge cases. I spent time making sure my comments were great, and then wrote a man page. I think they learned some important things about me, and on my end it was an easily paid investment.
We've been using something like this but centered around concurrency rather than networking. It works out great, you get to compare across many candidates how they handle the tricky parts, if they understand the standard vocabulary of concurrency, what their coding style is like and if they understand basic C++.
One thing we noticed was that there is small handful of mistakes that many candidates tend to make. The ones that note the existence of a mistake or work around them skillfully are typically the best.
One thing we noticed was that there is small handful of mistakes that many candidates tend to make. The ones that note the existence of a mistake or work around them skillfully are typically the best.
Yeah, something small is probably okay. I know that I've invested more than that preparing to interview for a new job (brushing up rusty skills, revising algorithms I don't use but interviewers love to ask about). The author talked about hiring people for weeks. I think that is too long.
This trial period is just a way of pushing the risk of a bad hiring decision back onto the candidate and allowing the manager to avoid the discomfort of actually firing someone (although telling the candidate that the project didn't work out is functionally the same thing).
It is tough firing people. But that doesn't mean it can be avoided.
And this is a good time to remind everyone: Firing an employee doesn't mean the employee failed. It means that everyone (including the company and everyone else involved) also failed. Such a failure in an organization should be a time of reflection for everyone; to think about and talk about what worked, what didn't, and what everyone can do better next time.
It is tough firing people. But that doesn't mean it can be avoided.
And this is a good time to remind everyone: Firing an employee doesn't mean the employee failed. It means that everyone (including the company and everyone else involved) also failed. Such a failure in an organization should be a time of reflection for everyone; to think about and talk about what worked, what didn't, and what everyone can do better next time.
The author suggests to pay a candidate for the trail project, which makes it a reasonable deal for the candidate.
To do something small, you don't need to quit a job, just dedicate few evenings and weekends and earn some extra cash.
This method can be beneficial for candidates, that can get to know potential employers much better, and avoid ones that look interesting only on job ads.
To do something small, you don't need to quit a job, just dedicate few evenings and weekends and earn some extra cash.
This method can be beneficial for candidates, that can get to know potential employers much better, and avoid ones that look interesting only on job ads.
The author specifically recommends hiring someone for weeks. Maybe he meant a couple of weekends, but it doesn't read that way. I agree that something small - evenings and weekends - might work. Of course, you and your team need to be prepared to come into work at those times too, in order to make this a bidirectional experience.
I don't know what it is about technical interview-style code puzzles, but I have always been terrible at them. Every time I go through a job hunt, I end up with at least one fabulous flameout where I just can't do something that would normally be very simple for me. It sucks. I have fantastic experience, good references, a degree from a top university, but you put me in a room with someone looking over my shoulder as I write code and I just turn into mush often enough that it makes passing a many round technical interview a statistical improbability.
The good news is that more and more companies (and especially the small ones that I like to work for) do the coding sample/test code thing. And every time I get one of those I absolutely crush it. I've never once been denied an offer when asked to write an offline code sample - and the offers have been from a number of the top startups in the game.
What does any of this mean? Well, in the last two companies I worked with I ended up rising to be the top technical person on my team. I've never had any kind of negative review of any sort. And in the last round this happened after I was denied an offer from two different companies because I couldn't code.
So I guess I'm that guy. I've been rejected from Google 3 different times because they keep begging me to interview based off my experience/internal recommendations, and then I interview and their interview/hazing process weeds me out. It would be nice if they woke up and realized that process isn't an effective filter, but I'm not holding my breath. In the meantime there's plenty of companies who will happily hire me after they see me do work that's actually relevant to the job. And when I interview people for my team I steer clear of trying to stump them under intense pressure, and instead ask for a mini project. I guess it doesn't feed the ego quite like breaking someone with a super awesome whiteboard question, but screening candidates more effectively makes up for that.
The good news is that more and more companies (and especially the small ones that I like to work for) do the coding sample/test code thing. And every time I get one of those I absolutely crush it. I've never once been denied an offer when asked to write an offline code sample - and the offers have been from a number of the top startups in the game.
What does any of this mean? Well, in the last two companies I worked with I ended up rising to be the top technical person on my team. I've never had any kind of negative review of any sort. And in the last round this happened after I was denied an offer from two different companies because I couldn't code.
So I guess I'm that guy. I've been rejected from Google 3 different times because they keep begging me to interview based off my experience/internal recommendations, and then I interview and their interview/hazing process weeds me out. It would be nice if they woke up and realized that process isn't an effective filter, but I'm not holding my breath. In the meantime there's plenty of companies who will happily hire me after they see me do work that's actually relevant to the job. And when I interview people for my team I steer clear of trying to stump them under intense pressure, and instead ask for a mini project. I guess it doesn't feed the ego quite like breaking someone with a super awesome whiteboard question, but screening candidates more effectively makes up for that.
Agreed, surprised this whole aspect isn't discussed much, much more.
There's obviously a massive difference between "can code proficiently" and "can code proficiently in an interview situation" and I would guess that a significant amount of even fizz-buzz fails are due to this, let alone complex algorithmic stuff.
When you're an interviewer doing many sessions it soon becomes very routine, to the point of tedium. Being an interviewee is on the other hand hugely stressful for a lot of people, and that just isn't conducive to good "on the spot" analytical thinking. And interviewers in their comfort zones completely forget this and don't factor it into performance (done so myself).
There's obviously a massive difference between "can code proficiently" and "can code proficiently in an interview situation" and I would guess that a significant amount of even fizz-buzz fails are due to this, let alone complex algorithmic stuff.
When you're an interviewer doing many sessions it soon becomes very routine, to the point of tedium. Being an interviewee is on the other hand hugely stressful for a lot of people, and that just isn't conducive to good "on the spot" analytical thinking. And interviewers in their comfort zones completely forget this and don't factor it into performance (done so myself).
This annoys me. Articles like this co-opt a term (technical interviews) and redefine it against a subset of things that qualify as being that term (brainteaser/quiz heavy technical interviews).
There is simply no way to reliably hire skilled people without at least attempting to assess their skills. For technical people this means technical interviews. And an interview process that includes a trial project... is still a technical interview.
This is distracting from the real issue. Most companies are really bad at scaling their interview policies/processes. Quiz/Brainteaser interviews are born from an effort to standardize evaluations across multiple interviewers and multiple candidates. So rather than focusing on developing an interview process that's focused on finding people that are a good fit, you're focusing on developing an interview process that's first easy for anyone on your side of the table to execute. Structure and process are good when they benefit the prior case, and stifling when they benefit the latter. In the end you wind up with a very expensive and completely repeatable noise generator that's of no benefit to anyone.
There is simply no way to reliably hire skilled people without at least attempting to assess their skills. For technical people this means technical interviews. And an interview process that includes a trial project... is still a technical interview.
This is distracting from the real issue. Most companies are really bad at scaling their interview policies/processes. Quiz/Brainteaser interviews are born from an effort to standardize evaluations across multiple interviewers and multiple candidates. So rather than focusing on developing an interview process that's focused on finding people that are a good fit, you're focusing on developing an interview process that's first easy for anyone on your side of the table to execute. Structure and process are good when they benefit the prior case, and stifling when they benefit the latter. In the end you wind up with a very expensive and completely repeatable noise generator that's of no benefit to anyone.
Giving someone a real project to do which they get paid for is a really bad idea. If the person is working for another company at the time then they likely have something their contract that specifically stops them doing work for someone else while still under employment. It's ground for a lawsuit against the employee.
It also means that as the interviewing company that you don't legally own the IP rights for the code you just put into production, which opens you up to a lawsuit as well.
And that's just for when you're hiring someone from another random company, if you're hiring someone from a competitor you've got a whole different layer of complexity in terms of NDA, non-compete, etc. violations.
And then you get into employment law and accounting complications. Who's declaring tax on the payment ? - if the candidate isn't setup for contracting work they'll need to do a bunch of work to sort out the taxes, if the company takes care of taxes than that might be taken as evidence of employment.
It's one of those ideas that might seem nice superficially but which will get you into a world of pain in the long term.
It also means that as the interviewing company that you don't legally own the IP rights for the code you just put into production, which opens you up to a lawsuit as well.
And that's just for when you're hiring someone from another random company, if you're hiring someone from a competitor you've got a whole different layer of complexity in terms of NDA, non-compete, etc. violations.
And then you get into employment law and accounting complications. Who's declaring tax on the payment ? - if the candidate isn't setup for contracting work they'll need to do a bunch of work to sort out the taxes, if the company takes care of taxes than that might be taken as evidence of employment.
It's one of those ideas that might seem nice superficially but which will get you into a world of pain in the long term.
You can't bar moonlighting in a work agreement in the US.
It doesn't stop things like IP ownership clauses which cover all work you do (including work outside of work hours).
It's the reason major open source projects require a waiver from your employer if you're contributing substantial code even if you're doing it in your spare time.
It's the reason major open source projects require a waiver from your employer if you're contributing substantial code even if you're doing it in your spare time.
AIUI, both legally and practically, you can!
Legally: a lot of graduate student employment contracts bar any other paid employment with clauses like your graduate student tuition deferment being lifted if it is discovered that you have a side job.
Practically: So, you moonlight for a competitor for a week and if your boss finds out that you added a shipping feature into someone elses product, and you didn't get the job..?
Legally: a lot of graduate student employment contracts bar any other paid employment with clauses like your graduate student tuition deferment being lifted if it is discovered that you have a side job.
Practically: So, you moonlight for a competitor for a week and if your boss finds out that you added a shipping feature into someone elses product, and you didn't get the job..?
those aren't legal statutes, those are contracts.
Just because a contract says something doesn't mean its enforceable. Many grad students moonlight with consulting work in computer science and other technical disciplines.
Just because a contract says something doesn't mean its enforceable. Many grad students moonlight with consulting work in computer science and other technical disciplines.
the law being on your side is a cold comfort when you're fired!
You cannot bar quitting in the U.S. You can bar multiple employers.
I haven't started applying for jobs yet (I'm still in undergrad) but I know for a fact that I would choke on those "brain teaser" questions, turn bright red and everything. Admittedly, there's just too much pressure (I hate looking stupid, especially in front of strangers, especially in front of male strangers--don't we all though?).
However, I'm pretty sure that I CAN do the following (even under pressure): Work through a problem rationally, work constructively with others, pay attention to detail without losing sight of the big picture, write code. And I would much rather focus on side projects than studying variations on Towers of Hanoi.
So this whole interviewing trend makes me want to cry out of relief.
However, I'm pretty sure that I CAN do the following (even under pressure): Work through a problem rationally, work constructively with others, pay attention to detail without losing sight of the big picture, write code. And I would much rather focus on side projects than studying variations on Towers of Hanoi.
So this whole interviewing trend makes me want to cry out of relief.
> (I hate looking stupid, especially in front of strangers, especially in front of male strangers--don't we all though?).
You should get over that. It is probably very important for your career (and life) that you not care about looking stupid in front of other people. It's going to happen (a LOT) in your professional career, dealing with it gracefully is one of the skills to have.
You should get over that. It is probably very important for your career (and life) that you not care about looking stupid in front of other people. It's going to happen (a LOT) in your professional career, dealing with it gracefully is one of the skills to have.
For many people, that's rather like asking them to "get over" their fear of heights; it's not very reasonable. It's the kind of thing you struggle through over the course of your lifetime, probably never totally solving it.
owch yeah. re-reading my words, they could be pretty harsh. I didn't mean for the glibness of my reply to imply that it would be easy - it's very hard! but from personal experience, it's very worth it.
> (I hate looking stupid, especially in front of strangers, especially in front of male strangers--don't we all though?).
I feel the same way. One thing that helped me cope with it was taking some improv classes. I still get nervous, but I can feed off the nervous energy instead of letting it get in my way.
I feel the same way. One thing that helped me cope with it was taking some improv classes. I still get nervous, but I can feed off the nervous energy instead of letting it get in my way.
[deleted](1)
Paid moonlighting projects are illegal for all us work visa holders but sometimes we wanna change jobs too. :(
I think this person misses the point of technical interview.
Technical interview is to assert the candidate actually has the skills and experience claimed in the resume.
When somebody is saying he or she has 20 years of C++ experience and 150 projects shipped yet he or she does not know what is virtual function - this means he or she lied on the resume. If you had not conducted a technical interview - you'd learned about this much later, at much greater cost. Chances are that somebody who lies on the resume is also good at BSing about "past projects" and "greatest achievements".
FizzBuzz or a general screen test won't help because people have different skills. If you are hiring somebody who writes Perl it's stupid to ask about C++ and vice versa.
Writing a test project - probably the best way to hire recent graduates. Probably the worst way to hire somebody who already has a job and you are desperate to poach, not really good at picking people who have several offers to choose from either.
When somebody is saying he or she has 20 years of C++ experience and 150 projects shipped yet he or she does not know what is virtual function - this means he or she lied on the resume. If you had not conducted a technical interview - you'd learned about this much later, at much greater cost. Chances are that somebody who lies on the resume is also good at BSing about "past projects" and "greatest achievements".
FizzBuzz or a general screen test won't help because people have different skills. If you are hiring somebody who writes Perl it's stupid to ask about C++ and vice versa.
Writing a test project - probably the best way to hire recent graduates. Probably the worst way to hire somebody who already has a job and you are desperate to poach, not really good at picking people who have several offers to choose from either.
The purpose of an interview is to see if the person is a good fit for the position at hand; looking at the "future". The past experience is a natural measure of future performance, provided the future projects at the new position are similar to the past projects. My experience [1] is that the past projects can often be significantly different from the future projects, and yet the candidate may be a great fit.
What does carry along with the person is still something about the way that person thinks, handles problems, basic aptitude, knowledge and understanding, etc. Fizzbuzz is a good measure of basic programming aptitude [2]. I have constantly used simple equivalents of Fizzbuzz for electrical engineering and have hired some great performers.
[1] I have conducted over 100 interviews. [2] C++ vs. Perl should not matter much since the question is about the basic algorithm.
What does carry along with the person is still something about the way that person thinks, handles problems, basic aptitude, knowledge and understanding, etc. Fizzbuzz is a good measure of basic programming aptitude [2]. I have constantly used simple equivalents of Fizzbuzz for electrical engineering and have hired some great performers.
[1] I have conducted over 100 interviews. [2] C++ vs. Perl should not matter much since the question is about the basic algorithm.
>The purpose of an interview is to see if the person is a good fit for the position at hand
This depends on the position. If the position requires an MS degree and 5 years of experience you'd need to spend couple of weeks testing for the whole corpus of knowledge, that these requirements demand. It's quite expensive and few candidates are willing to spend this much time. The only practical way is to go with the credentials the person already has.
If you are looking for somebody to write websites using frameworks then, indeed, you can test all required skills in a day of technical interviews.
The handling of problems would be nice to know but, unfortunately, is hard to test. Somebody who just flew into the interview and is confronted by a bunch of strangers is not in the best shape to handle problems. If he can even in these conditions - good, now you know that this person can work under stress. Successfully solving simple puzzles though does not show anything more than this - you are likely looking for somebody to solve much harder problems that require weeks and months of research. Neither does a negative result mean much - this is most likely a result of stress.
So the brain teasers and "how many sorting algos you can write on a white board in 30 mins" type of questions, in my opinion, are useless. Asking concrete questions in the domain of the candidates knowledge on the other hand, lets you quickly verify much wider set of skills.
This depends on the position. If the position requires an MS degree and 5 years of experience you'd need to spend couple of weeks testing for the whole corpus of knowledge, that these requirements demand. It's quite expensive and few candidates are willing to spend this much time. The only practical way is to go with the credentials the person already has.
If you are looking for somebody to write websites using frameworks then, indeed, you can test all required skills in a day of technical interviews.
The handling of problems would be nice to know but, unfortunately, is hard to test. Somebody who just flew into the interview and is confronted by a bunch of strangers is not in the best shape to handle problems. If he can even in these conditions - good, now you know that this person can work under stress. Successfully solving simple puzzles though does not show anything more than this - you are likely looking for somebody to solve much harder problems that require weeks and months of research. Neither does a negative result mean much - this is most likely a result of stress.
So the brain teasers and "how many sorting algos you can write on a white board in 30 mins" type of questions, in my opinion, are useless. Asking concrete questions in the domain of the candidates knowledge on the other hand, lets you quickly verify much wider set of skills.
I am largely in agreement on what you wrote here.
Whilst I'd love to have them do a paid project for me that will actually ship, I work in a part of the programming world where a project easily runs a decade from first draft of requirements to entering maintenance. This sort of suggestion always seems to come from the world of web apps or SaaS or whatever else is hip at the moment.
I also can't imagine what HR would say if I told them I wanted to hire someone for a week just to see how they get on. "Well, then your HR system is broken and you should totally hack it to make it better." I'll get right on that, just as soon as I become Vice-President of HR. They'd tell me to do my job and hire people who can do what they're hired for.
Where this is going is the ripost that this kind of suggestion always comes from the small part of the industry where it might be practical; the are are a great many programming jobs in companies where it really, really isn't.
I also can't imagine what HR would say if I told them I wanted to hire someone for a week just to see how they get on. "Well, then your HR system is broken and you should totally hack it to make it better." I'll get right on that, just as soon as I become Vice-President of HR. They'd tell me to do my job and hire people who can do what they're hired for.
Where this is going is the ripost that this kind of suggestion always comes from the small part of the industry where it might be practical; the are are a great many programming jobs in companies where it really, really isn't.
I guess I am such a terrible developer, that I too can't imagine a project that is useful that can ship after just a week's work (of probably part time while working at my real job).
I did an online coding interview a few months back, and after working with some dude for 20 minutes on some program he asked me if it was ready to ship. And I was "uh, no, of course not" since we had done no testing of any sort apart from mentally running through the code. "Well, make it ready to ship!" I don't think he understood. The two of us certainly didn't agree on what software engineering means.
I did an online coding interview a few months back, and after working with some dude for 20 minutes on some program he asked me if it was ready to ship. And I was "uh, no, of course not" since we had done no testing of any sort apart from mentally running through the code. "Well, make it ready to ship!" I don't think he understood. The two of us certainly didn't agree on what software engineering means.
Although I like some aspects of the trial project idea, I agree that having something shipped to production after one week of work on a completely new project is in most cases unreasonable.
I see that hyperbolic headlines are going strong, however.
This is from the same organization that proclaims your startup dead if it hasn't raised any new venture capital in the last 12 months. Hyperbolic is all they do.
I interviewed with one of the big Internet companies not too long ago and found the process pretty refreshing and quite unlike what I'd so often read about.
It opened up with a sort of multi-faceted FizzBuzz followed by long "behavioral" discussion like one of the references refers to [0].
The audition process doesn't make any sense to me. It's essentially a short duration temp-to-hire. Great people tend to be employed or at least not idle, unemployed people are looking for jobs, not 1 week gigs.
0: http://www.nytimes.com/2013/06/20/business/in-head-hunting-b...
It opened up with a sort of multi-faceted FizzBuzz followed by long "behavioral" discussion like one of the references refers to [0].
The audition process doesn't make any sense to me. It's essentially a short duration temp-to-hire. Great people tend to be employed or at least not idle, unemployed people are looking for jobs, not 1 week gigs.
0: http://www.nytimes.com/2013/06/20/business/in-head-hunting-b...
I think whats always been hard for me is dealing with the problem I'm trying to solve and all the anxiety happening at the same time. I feel totally comfortable writing code, but when 80% of brain is dedicated to flight-or-fight response it makes it extremely hard to answer technical questions and pass the interview.
My favorite interview that I took part in and got hired was when I was given 2 hours to pair program a project with the interviewer. So I really like the concepts in this post. Anything to get the developer to calm down and feel at home.
My favorite interview that I took part in and got hired was when I was given 2 hours to pair program a project with the interviewer. So I really like the concepts in this post. Anything to get the developer to calm down and feel at home.
Two common kinds of reactions by people who fail tech interviews at Google, Facebook, etc:
1. Think how to improve and practice for the next interview.
2. Bitch about how technical interviews are unfair, biased, miss the best people and ... OMG - DEAD!!!
1. Think how to improve and practice for the next interview.
2. Bitch about how technical interviews are unfair, biased, miss the best people and ... OMG - DEAD!!!
3. Get a job at another company that doesn't hire via silly hazing rituals. Preferably one that isn't past its heyday like the two you mentioned.
What is so bloody hard about just putting a laptop in front of someone, ask them to do a bunch of basic but real programming tasks, including internet access and see what happens? The overall jist I get out of people have been pretty accurate the months and years I have worked with them.
I dropped the ball on one employer because the audition project took too long.
I dropped the ball on one employer because the audition project took too long.
Isn't there some real logistical problems with that. For example I am used to an particular keyboard and typing on a laptop keyboard is big issue for me.
When I do the laptop interview, I bring a mouse I ask them if they want a keyboard, a different mouse, water or anything else. I even offer to leave the room if they are freaking out. Only one guy has even used the mouse.
Overall although, I think people are far more comfortable with typing on a 15" macbook pro vs scribbling something on a whiteboard.
Overall although, I think people are far more comfortable with typing on a 15" macbook pro vs scribbling something on a whiteboard.
If you simply type slower (or with a higher error rate) on a different keyboard, I suspect no one would care. If it's a medical condition, I suspect a reasonable company that wishes to avoid probably valid lawsuits would let you bring in your own keyboard.
It not about others caring, rather it will a major distraction for me and we are back with a uneasy interviewee (which was the problem with whiteboard coding).
I've done interviews before. I think that there are still many older people, and young people who do not have accounts on sites like github, they have nothing to show. But may be reasonable candidates to hire because of their technical talent. I think it's appropriate to ask questions to an interviewee based on the domain knowledge you'd expect them to have during an interview. These involve little things that are very typical of day-to-day work. These are the type of things I'd ask over the phone or email. Once you determine they are not blowing smoke up your rear, ask them to come in for a more formal person-to-person interview.
I then ask them a fairly broad open-ended set of questions to see how they describe the process of solving said problem.
This usually involves talking about tools, their process, their libraries of choice(something more generic here for other fields) and generally helps to assess their personable skills.
If I like them, I will ask them if they have any personal projects they want to share or something they are enthusiastic about in their domain of knowledge.
I think the biggest thing is that a lot of companies doing technical interviewing only reserve a small period of time to determine their new candidate. For people long in the industry, their resume should speak for itself. For younger people, it's kind of like, would you sleep with a woman you just met? How does that experience usually turn out? It's a 50/50 or worse chance you end up with a nightmare. Spend some time to interview your candidates, or end up with the one-night-stand.
I then ask them a fairly broad open-ended set of questions to see how they describe the process of solving said problem.
This usually involves talking about tools, their process, their libraries of choice(something more generic here for other fields) and generally helps to assess their personable skills.
If I like them, I will ask them if they have any personal projects they want to share or something they are enthusiastic about in their domain of knowledge.
I think the biggest thing is that a lot of companies doing technical interviewing only reserve a small period of time to determine their new candidate. For people long in the industry, their resume should speak for itself. For younger people, it's kind of like, would you sleep with a woman you just met? How does that experience usually turn out? It's a 50/50 or worse chance you end up with a nightmare. Spend some time to interview your candidates, or end up with the one-night-stand.
This is funny because they've adopted the same interviewing style that large bureaucratic organizations have been using for at least a decade, even for technical positions. I went through a behavioral interview at Boeing, and then another at a large federal agency (engineering job). Technical aptitude is usually clobbered by the inability to work with people.
This article doesn't mention the best interview method of them all: the presentation interview. Have the interviewee prepare a 15 minute presentation of a technical topic of the interviewee's choice. At the end of that presentation the team should have a general idea of the person's thinking speed/experience/communications skills. Basically all you'd ever need to know about the person.
In my version of a perfect world, each company would have a "speaking hour" designated once per week. Each speaking hour would feature two or three speakers, each with 15 minutes to speak about something interesting they've worked on. The idea being if you wanted to work for a company, you'd apply to speak at their next "speaking hour". If you show up and give your talk and the audience approves your talk, then its understood you're hired.
In my version of a perfect world, each company would have a "speaking hour" designated once per week. Each speaking hour would feature two or three speakers, each with 15 minutes to speak about something interesting they've worked on. The idea being if you wanted to work for a company, you'd apply to speak at their next "speaking hour". If you show up and give your talk and the audience approves your talk, then its understood you're hired.
[deleted]
[deleted]
I like the idea of using the interview as a screening process, just to make sure the candidate didn't lie on their resume, and that they are a socially competent person who can survive in a team environment, or better yet, improve the team.
But, you should also carefully examine the past code that the candidate has written (that he can legally disclose) and also design a programming test for candidates to complete.
But, you should also carefully examine the past code that the candidate has written (that he can legally disclose) and also design a programming test for candidates to complete.
One good observation from the article is that more simple questions is better than a few harder ones. Not much besides this.
My observation is that recruitment is only one part of the problem. The other part is that companies do very poor job at assessing skills during employment and loose/fire randomly. This seems to be much more important problem for top companies leading to slow degradation of skills over time.
My observation is that recruitment is only one part of the problem. The other part is that companies do very poor job at assessing skills during employment and loose/fire randomly. This seems to be much more important problem for top companies leading to slow degradation of skills over time.
We talk a little about how each one of us at Techendo does our hiring. It might offer a bit of insight on what works and what doesn't. http://www.youtube.com/watch?v=xHunZeWOrjc
How about explaining your insights in ten sentences or so in this thread instead of just linking to some video that only few people would actually go watch just like that?
Reword the title as "Is the technical interview dead?" and you have your answer: it's always "No".
So, finally, "show us your code/poetry/papers" won?)
The "audition project" is a trend story with, from what I can see, very little empirical evidence to back it up. When the most notable example of a company routinely employing audition is Treehouse --- all due respect to Treehouse, but still --- that's a red flag.
The bigger red flag is that the paid audition project method has obvious flaws. The top 20% of software developers are almost uniformly employed. Prospective new employers for these people court them actively; in fact, the problem of companies luring top developers out of other companies is so challenging that large SFBA tech companies have entered into illegal compacts to stifle the practice. Companies are so hard-up for talent that they'll "buy" startups just to get access to their teams.
In this environment, why would a top developer, who has their choice among tens of different high-paying interesting jobs, moonlight for a prospective new employer just so they can make sure the relationship's going to work?
Most technical interviews are terribly flawed. They aren't standardized and they aren't rigorous so you can't compare candidates on any apples-apples basis and you can't correlate them to job performance to make them more predictive. Most of the developers tasked with conducting them suck at interviewing; many use interviews as a sort of hazing ritual, and most use them as opportunity to project their own subjective views about how software should be built onto candidates. Many technical interviews are trivia games punctuated by awkward attempts at working through code on a whiteboard or a piece of paper in a high-pressure environment.
The solution to this problem is to improve technical interviews. It's not to pretend that the whole market for devs has suddenly embraced temp- to- perm hiring. It hasn't. It's getting harder to find good developers, not easier, and the notion that companies have the luxury of inflicting "audition projects" on candidates is counter to reality.