I never said radiology didn't produce biased results, just that we don't know the race of our patients most of the time.
There are lots of ways bias can still occur, like in who gets referred for scans, when they get referred, what the referrer writes on the request form, how the technologist takes the images (I could tell you some horribly racist stories about a few ultrasonographers I've worked with), and so on.
And all of this is based on previous work that AI produces bias (when trained on these datasets). If it was useful differences that drove AI learning about race, the models would not produce disparities. We went looking for how it is interacting with race because we already knew it was producing unacceptable outcomes. The big news here is that it is so easy to learn race that this effect is almost certainly not isolated to the systems tested so far.
Hi, author here. I'll try to answer here with two points.
1) race is not provided explicitly or intentionally during training, but medical practice is biased so it is reasonable to assume there is a signal in the data.
2) we know there must be a signal, since AI models learn it. The optimisation process should only discover features that correlate with the labels, but we see the ability to predict race in models trained to look for diseases like pneumonia (which do not appear different for people from different racial groups).
I don't actually think imagenet is anywhere near as susceptible to crowd based overfitting as most kaggle competitions, but I don't actually think that paper falsifies the claim that it is.
That paper shows that imagenet classifiers retain their ranking on data drawn from the same distribution as the training set. That isn't even close to the same thing as generalising to real world data. Medical AI shows us over and over again that truly out of distribution unseen data (external validation) is a completely different challenge to simply drawing multiple test sets from your home clinic.
Again, I don't actually think imagenet was as problematic as other competitions, but there is better evidence for that (not the least of which is that for the first half of imagenet's life, the differences in models were large, the number of tests was cumulatively fairly small, and the test set was huge: ie what I wrote supports imagenet as fairly reliable).
All I can really say is that my usual readers understand that I am pro-ML, in fact I'm probably more hung go about the potential of deep learning than many of my compatriots.
I've fallen victim of getting a Twitter bump, and assuming that people know I'm not anti-ML.
The blog post is meant to be educational, not argumentative. Since it has got wider exposure I'll do a follow up to clarify my position on imagenet.
Hi, author here. There are a range of ways the estimates can be improved, although many require data that isn't available. The main point is that having a ballpark idea of how reliable your results are is good, and you can achieve that with this sort of simple napkin maths.
No statistician would do what I did for a formal publication, but I think what I did gets the point across.
Hi, author here. I didn't actually mean to suggest that the last 5 years of performance improvement could be spurious. That clearly isn't true. I use resnets/densenets etc in my day to day work!
What the picture was trying to say is that, within a given year, the "winner" becomes less likely to be truly better than the second place team. Alexnet was clearly better than the alternative, even with Bonferroni adjusted significance thresholds. Less so by 2016/17.
I'm writing a follow up on imagenet in particular to address some of the nuance. It is very clearly not a representative example of ML competitions, but the same effects still apply to some extent (imo).
Hey, thanks for liking the article. I was just looking at my blog stats and they had a bit of spike when you wrote this.
I'm just getting down to writing some posts about the big question the New Yorker article introduces, but doesn't really make any real progress with: will machines actually replace doctors.
Hi, I'm the author of the blogpost (didn't expect to see it here!) and I largely agree with you.
I think I would say that the use of private datasets isn't actually a problem though, the problem is that the methods and analysis in many of the papers is horribly flawed. I could reasonably trust a paper with good methodology - all of medical research is on private data essentially. But the majority is, as you say, pure hype.
There are lots of ways bias can still occur, like in who gets referred for scans, when they get referred, what the referrer writes on the request form, how the technologist takes the images (I could tell you some horribly racist stories about a few ultrasonographers I've worked with), and so on.
And all of this is based on previous work that AI produces bias (when trained on these datasets). If it was useful differences that drove AI learning about race, the models would not produce disparities. We went looking for how it is interacting with race because we already knew it was producing unacceptable outcomes. The big news here is that it is so easy to learn race that this effect is almost certainly not isolated to the systems tested so far.