That's true of any hiring system though: how do you gather data on the job performance of the people who don't pass a resume screen? A coffee date? A phone screen? Missing counterfactuals everywhere.
To run a full analysis, you need to hire people randomly - both people who fail and pass your hiring process - and assess their subsequent performance. This never happens in real life.
However, you can still run an informative statistical analysis based on the variability in interview scores and performance scores. For example, the people who scored 5/5 on the interview should perform better than the ones who scored 4/5.
Here's the NYT article where Laszlo mentions the "random mess" they found when they used the interviewer as the predictor and then subsequently mentions how they use structured interviews now instead: http://www.nytimes.com/2013/06/20/business/in-head-hunting-b...
My point is, it doesn't matter who does the interviewing - looking for people who are "good interviewers" is a unicorn chase because the vast majority of people don't have a magical ability to guess who's going to be a good performer pre-hire. Unfortunately, Amazon is making the same mistake with their "bar raisers": http://firstround.com/article/Mechanize-Your-Hiring-Process-...
Geez, you engineers don't seem to like to learn from each other.
From the NYT's article quoting Laszlo Bock: "Years ago, we did a study to determine whether anyone at Google is particularly good at hiring."
Google made the classic mistake of using the interviewer as the predictor of interest. Meanwhile, decades of research has shown it's not who does the asking but how and what you ask. For example, structured interviews are about 50% more predictive of job performance (r = .51) compared to unstructured interviews (i.e., the way almost everyone interviews; r = .38).
So why did Google run this experiment when they had an army of Industrial/Organizational Psychologists who already knew the research? They wanted to collect their own data and test the theories on their own employees, which I encourage and applaud.
To run a full analysis, you need to hire people randomly - both people who fail and pass your hiring process - and assess their subsequent performance. This never happens in real life.
However, you can still run an informative statistical analysis based on the variability in interview scores and performance scores. For example, the people who scored 5/5 on the interview should perform better than the ones who scored 4/5.