Machine Learning and Human Bias: An Uneasy Pair(techcrunch.com)
techcrunch.com
Machine Learning and Human Bias: An Uneasy Pair
http://techcrunch.com/2015/08/02/machine-learning-and-human-bias-an-uneasy-pair/
4 comments
I think it's a much more subtle point than that, and he does make it rather directly:
* If police departments are racially biased * and the Heat List algorithm heavily factors in associations * and most people associate with others from their own race * then won't the Heat List disproportionately output people from a certain race? * Won't this then result in increased policing & suspicion of these communities?
In other words, the point is: human biases can be a seed issue that machine learning then positive-feedback-loops out of control
* If police departments are racially biased * and the Heat List algorithm heavily factors in associations * and most people associate with others from their own race * then won't the Heat List disproportionately output people from a certain race? * Won't this then result in increased policing & suspicion of these communities?
In other words, the point is: human biases can be a seed issue that machine learning then positive-feedback-loops out of control
then won't the Heat List disproportionately output people from a certain race?
This will only happen if that certain race commits more crimes (in the training data). If you take race out of a statistical predictor designed to learn crime, but race is a good predictor of crime, then the predictor might learn race at an intermediate step.
Now there are statistical issues one might run into - e.g., early overfitting of what is essentially a bandit algorithm, and unaccounted for feedback between training data and system outputs. But at least the way I'm reading the article, it isn't calling for more and better math (which would be the solution to the problems you describe).
This will only happen if that certain race commits more crimes (in the training data). If you take race out of a statistical predictor designed to learn crime, but race is a good predictor of crime, then the predictor might learn race at an intermediate step.
Now there are statistical issues one might run into - e.g., early overfitting of what is essentially a bandit algorithm, and unaccounted for feedback between training data and system outputs. But at least the way I'm reading the article, it isn't calling for more and better math (which would be the solution to the problems you describe).
I think you're missing the first point in both my summary and in the article: It does not need to be the case that "a certain race commits more crimes". It can, instead, just be the case that a certain race is arrested for committing more crimes, despite the equal rates across races of the actual criminal behavior.
For instance: it's a well recognized fact[1] that blacks and whites use and deal marijuana at the same rate, but blacks are arrested for it in far larger volume. So, if this data and other similar data sets are the seed in a machine learning algorithm, then algorithms like the Heat List will output racially biased data.
[1] https://www.aclu.org/files/assets/aclu-thewaronmarijuana-rel...
For instance: it's a well recognized fact[1] that blacks and whites use and deal marijuana at the same rate, but blacks are arrested for it in far larger volume. So, if this data and other similar data sets are the seed in a machine learning algorithm, then algorithms like the Heat List will output racially biased data.
[1] https://www.aclu.org/files/assets/aclu-thewaronmarijuana-rel...
I didn't miss that. As I said: Now there are statistical issues one might run into - e.g., early overfitting of what is essentially a bandit algorithm, and unaccounted for feedback between training data and system outputs.
There is nothing fundamental about machine learning that says seed data like this will give biased outputs - many algorithms do have this problem (it's a difficult one to deal with), but it's not fundamental.
I certainly didn't get the impression from the article that it was advocating for algorithms which are less sensitive to these errors. Among other things, that's far less of a conversation that "we have to talk about", but far more of a conversation that some stats geeks have to talk about. These are also far less of an "ethical" problem (as the article asserts) and far more of a technical one.
But maybe I misread.
There is nothing fundamental about machine learning that says seed data like this will give biased outputs - many algorithms do have this problem (it's a difficult one to deal with), but it's not fundamental.
I certainly didn't get the impression from the article that it was advocating for algorithms which are less sensitive to these errors. Among other things, that's far less of a conversation that "we have to talk about", but far more of a conversation that some stats geeks have to talk about. These are also far less of an "ethical" problem (as the article asserts) and far more of a technical one.
But maybe I misread.
It depends a lot on the reporting rate for the kind of crime being monitored; things like drug use - where enforcement is almost entirely dependent on police action - will absolutely be subject to this kind of positive bias. On the other hand, violent crimes, like shootings, are events that are (usually) observed regardless of police involvement.
There's been some pretty interesting work on this in Richmond, CA, up north of Berkeley: http://www.thisamericanlife.org/radio-archives/episode/555/t... Basically, it was noticed that a small cluster of people were involved in most of the violent crime, and started interventions to make that cluster less likely to engage in violent crime.
There's been some pretty interesting work on this in Richmond, CA, up north of Berkeley: http://www.thisamericanlife.org/radio-archives/episode/555/t... Basically, it was noticed that a small cluster of people were involved in most of the violent crime, and started interventions to make that cluster less likely to engage in violent crime.
Won't this then result in increased policing & suspicion of these communities?
Increased policing should lead to lower crime levels in these areas and communities reversing the association. Unless you don't believe that policing works but then why bother with heat lists at all, no matter how accurate?
Increased policing should lead to lower crime levels in these areas and communities reversing the association. Unless you don't believe that policing works but then why bother with heat lists at all, no matter how accurate?
I also felt the article could have tried to get to the point more directly.
My initial reading of the article was that they were hand-wringing about whether the feature-set predicting crime could double as a feature-set for predicting race, and whether we should discount the algorithms as a result. To which my reaction was 'meh.' Just don't explicitly use race as a feature, and make sure your feature set will find suspicious people regardless of their race.
Feedback loops, on the other hand, are a completely legitimate concern, and something that should be asked about and controlled for in these kinds of analyses.
My initial reading of the article was that they were hand-wringing about whether the feature-set predicting crime could double as a feature-set for predicting race, and whether we should discount the algorithms as a result. To which my reaction was 'meh.' Just don't explicitly use race as a feature, and make sure your feature set will find suspicious people regardless of their race.
Feedback loops, on the other hand, are a completely legitimate concern, and something that should be asked about and controlled for in these kinds of analyses.
I think using machine learning for this purpose is equivalent to using stereotypes or intuition to make such decisions.
The reason people don't want to use stereotypes is that we might end up in a positive feedback loop when stereotyped people get arrested for the things they were expected to do, not because stereotypes aren't effective.
If it is illegal to do stop and frisk, we should not be doing machine learning where features come from intuition in the first place.
The reason people don't want to use stereotypes is that we might end up in a positive feedback loop when stereotyped people get arrested for the things they were expected to do, not because stereotypes aren't effective.
If it is illegal to do stop and frisk, we should not be doing machine learning where features come from intuition in the first place.
> What if machine learning systems come up with results that agree with "evil" stereotypes and "biases"?
This is already the case, and shouldn't be surprising. That is why the legal system can veto popular opinion, to protect minorities from the prejudice of the majority. The problem is that many people still don't realize that algorithms reinforce popular prejudice and institutionalized bias against the uneducated and the poor.
This is already the case, and shouldn't be surprising. That is why the legal system can veto popular opinion, to protect minorities from the prejudice of the majority. The problem is that many people still don't realize that algorithms reinforce popular prejudice and institutionalized bias against the uneducated and the poor.
If computer systems are getting the wrong conclusion, that's a boring philosophical question. Find better SVM params/neural network architecture/etc and you'll fix it. But what if the only way to eliminate "institutionalized bias" from the system is to add statistical bias? What happens if "popular prejudice" isn't actually prejudice but merely a useful prior, and one which non-racist [1] machine learning systems faithfully reproduce?
[1] I'm using "racist" in the classical sense of f(x.copy(race=A)) != f.copy(x.copy(race=B)), not in the modern sense of "any probabilistic cause for unequal statistical outcomes".
[1] I'm using "racist" in the classical sense of f(x.copy(race=A)) != f.copy(x.copy(race=B)), not in the modern sense of "any probabilistic cause for unequal statistical outcomes".
What if the data used to teach the machine is severely biased?
I'm reminded of the classic Charles Baggage quote:
On two occasions I have been asked, "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" ... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
I'd argue for a pretty high level of transparency in the process - I would like to see whatever classifiers being used open-sourced, for example. And I'd want to know where people are drawing the training data from.
But the nice thing is that the tech industry has a large population of people very sympathetic to transparency, and with a history of a culture supporting it. Quite frankly, I think the legal community has a lot more to learn from the open-source community than the other way around.
On two occasions I have been asked, "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" ... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
I'd argue for a pretty high level of transparency in the process - I would like to see whatever classifiers being used open-sourced, for example. And I'd want to know where people are drawing the training data from.
But the nice thing is that the tech industry has a large population of people very sympathetic to transparency, and with a history of a culture supporting it. Quite frankly, I think the legal community has a lot more to learn from the open-source community than the other way around.
Summarizing the complexities of human behavior using models appears as an unpleasant echo from the past. Statistical pioneers Karl Pearson and Francis Galton were strong proponents of social Darwinism aka scientific racism [1, 2].
The biggest problem with an observational approach to aggregate human behavior is that it generally ignores internal structure and makes hasty judgements based on mere appearance.
[1] https://en.wikipedia.org/wiki/Karl_Pearson#Politic
[2] https://en.wikipedia.org/wiki/Francis_Galton#Heredity_and_eu...
The biggest problem with an observational approach to aggregate human behavior is that it generally ignores internal structure and makes hasty judgements based on mere appearance.
[1] https://en.wikipedia.org/wiki/Karl_Pearson#Politic
[2] https://en.wikipedia.org/wiki/Francis_Galton#Heredity_and_eu...
So was Ronald Fisher. He was a big proponent of eugenics.
While at Harvard Law School, Barrack Obama argued that economic status is better indicator of crime than race (can't find the cite, as there is more written about Obama than by him, at HLS).
If using an Artificial Neural Network, would this mean that race should be downstream from economic status? Maybe race should be a hidden layer? What should be the input nodes? Is there a way to automatically create input nodes?
Perhaps finding the better inputs could lead to proactive measures? For example, let's say there several inputs are: adolescence, public schools, evenings, and free time. That might lead to keeping schools open late for extracurricular activities.
If using an Artificial Neural Network, would this mean that race should be downstream from economic status? Maybe race should be a hidden layer? What should be the input nodes? Is there a way to automatically create input nodes?
Perhaps finding the better inputs could lead to proactive measures? For example, let's say there several inputs are: adolescence, public schools, evenings, and free time. That might lead to keeping schools open late for extracurricular activities.
Thinking about things that way comes across very much like rationalizing the conclusions you expect, and building a machine that will just agree with you.
Yes, I suppose so if I was the one choosing the inputs. That's why I rather the machine to find the inputs automatically. But, then again, I supposed there is the bias of the training set.
What if machine learning systems come up with results that agree with "evil" stereotypes and "biases"? What if machine learning systems discover that socially unpleasant stereotypes are actually accurate predictors of reality?
Modern moral philosophy has taken an easy copout in the past. It asserts, without proof, that various positive claims are false, and therefore will yield bad decisions if used. We also believe it would be evil to use them to make decisions. The question we need to address is what moral claims can we make which are independent of positive claims?