Computer personality judgments more accurate than those made by humans: study(pnas.org)
pnas.org
Computer personality judgments more accurate than those made by humans: study
http://www.pnas.org/content/112/4/1036
12 comments
Haha, this headline is not a little bit misleading, it borders on the absurd! One might as well do a study on how "Computers are better at enjoying food than humans" but it is interesting, to use machine learning to see what we can gather purely from social media footprints versus what people see through one anothers' eyes. It makes sense, their abstract, when it says that the computer aided analysis is "more accurate" only insofar as many people will simply rely on their echo chamber for value assessments, and will therefore miss out on a key ~pseudo-objective viewpoint grounded in a more global image, which is what the computers can account for, and what your close friends cannot.
"This study compares the accuracy of human and computer-based personality judgments, using a sample of 86,220 volunteers... Computer predictions based on a generic digital footprint (Facebook Likes) are more accurate (r = 0.56) than those made by the participants’ Facebook friends using a personality questionnaire (r = 0.49)"
The fact that this is even close (let alone higher for computers) is pretty incredible. It is also quite scary for anyone with a public profile and speaks to the potential power that an organization with bad intentions or ulterior motives could have (ex: Cambridge Analytica).
The fact that this is even close (let alone higher for computers) is pretty incredible. It is also quite scary for anyone with a public profile and speaks to the potential power that an organization with bad intentions or ulterior motives could have (ex: Cambridge Analytica).
If you think about the Foundations series, it was basically about this: studying the psychology of masses of people and predicting (and influencing) their behavior.
Wow, that's really great to read. I did not get this idea from the first Foundation much, although it was a little more prevalent in the second book. I got something rather different from Foundation: the idea that one with omniscience shares what is timely and relevant for the benefit of all.
Hari Seldon's psychohistory brings back fond memories.
Wikipedia link: https://en.wikipedia.org/wiki/Psychohistory_(fictional)
Wikipedia link: https://en.wikipedia.org/wiki/Psychohistory_(fictional)
How do you feel about all the recommandations algorithms out there?
This 2015 paper is in the news now because of the "information operations" campaign that abused Facebook data to influence the 2016 election.
Authors of this paper communicated with the people behind the information operations campaign [0, 1], but it's not clear to me how closely they collaborated.
[0] https://motherboard.vice.com/en_us/article/mg9vvn/how-our-li...
[1] https://www.theguardian.com/news/2018/mar/17/data-war-whistl...
Authors of this paper communicated with the people behind the information operations campaign [0, 1], but it's not clear to me how closely they collaborated.
[0] https://motherboard.vice.com/en_us/article/mg9vvn/how-our-li...
[1] https://www.theguardian.com/news/2018/mar/17/data-war-whistl...
> Our findings highlight that people’s personalities can be predicted automatically and without involving human social-cognitive skills.
I had noticed with with Netflix. After telling it enough movies I liked (over say 100) it pretty much figured me out and it knew me better than I knew myself. In other words, I'd see a movie description, watch the trailer and think "Nah, I wouldn't like it". But Netflix was telling me I'd like it, quite often after I watched it I did end up liking it.
On another note, I always tried to steer away from taking personality tests. I have a strange belief that once I know the "official" results it would somehow restrict my what I would attempt later in life. I'd think "Oh well your personality officially doesn't match, you shouldn't try that".
I had noticed with with Netflix. After telling it enough movies I liked (over say 100) it pretty much figured me out and it knew me better than I knew myself. In other words, I'd see a movie description, watch the trailer and think "Nah, I wouldn't like it". But Netflix was telling me I'd like it, quite often after I watched it I did end up liking it.
On another note, I always tried to steer away from taking personality tests. I have a strange belief that once I know the "official" results it would somehow restrict my what I would attempt later in life. I'd think "Oh well your personality officially doesn't match, you shouldn't try that".
But Netflix has way more information than you do about the movie before you've seen it, namely you get a 30 second trailer, and it gets how many million people liked it. That doesn't show it is better than you, just that it has a huge information advantage.
One of the surprises with the Netflix prize contest was seeing how predictable people are.
You don't need 100 movies. 95% of the time, 6-10 movie ratings will be enough to figure out how you'd rate any other movie.
You don't need 100 movies. 95% of the time, 6-10 movie ratings will be enough to figure out how you'd rate any other movie.
For some reason, I feel like their control wasn't the best thing to compare to. People filling out personality questionnaires might not compare to looking at facebook likes. May be a better control would be letting humans look at the facebook likes themselves and compare their prediction based on that to that of computers looking at the set of facebook likes.
If they want to stick to the questionnaire, then train the computer model on the questionnaire too and let's see who wins. Giving one side more powerful data and giving the control just a questionnaire is like giving a robot a machine gun and a human a knife and being surprised who wins.
If they want to stick to the questionnaire, then train the computer model on the questionnaire too and let's see who wins. Giving one side more powerful data and giving the control just a questionnaire is like giving a robot a machine gun and a human a knife and being surprised who wins.
Have a feeling that if one used something deeper than just lasso regression, much better results could have been obtained. This is very similar to a recommender system, or a text classification problem, and there much better than lasso can be done. The data is sort of open, in principle, btw.
But nevermind what can be predicted from the likes. They say that coworkers have r=0.27, while spouse has r=0.58. This looks like a serious problem with the test more than anything else.
This may be interpreted as "cowokers don't know you (and don't care...)" kind of stuff. But coworkers often have a comparable amount of contact with a person vs spouse. Alternatively, one could also interpret this as there is no "the one true personality", but different personalities in different environments (at least when we define personality as a result of a test). And so coworkers are not wrong, they just see a different picture.
And there might be other possibilities.
In fact, one should try to predict the coworker's (and spuse's, ect...) scores of a person from person's likes, in addition to trying to predict person's own assessment.
But nevermind what can be predicted from the likes. They say that coworkers have r=0.27, while spouse has r=0.58. This looks like a serious problem with the test more than anything else.
This may be interpreted as "cowokers don't know you (and don't care...)" kind of stuff. But coworkers often have a comparable amount of contact with a person vs spouse. Alternatively, one could also interpret this as there is no "the one true personality", but different personalities in different environments (at least when we define personality as a result of a test). And so coworkers are not wrong, they just see a different picture.
And there might be other possibilities.
In fact, one should try to predict the coworker's (and spuse's, ect...) scores of a person from person's likes, in addition to trying to predict person's own assessment.
Coworkers often have a comparable amount of contact with a person vs spouse.
I understand what you mean. But the personality properties like agreeableness, extroversion, openness, should reveal themselves through behavior over time. Otherwise, what's the point in them? And, say, hi-tech work seems to provide enough opportunities, say over a year, to reveal all of them. This is what I meant by "contact".
Besides, have you not heard the "we do not spend enough time together" idea from a spouse? Not to glorify it, nothing good about it, but it seems to be the reality for more people than it should be.
Besides, have you not heard the "we do not spend enough time together" idea from a spouse? Not to glorify it, nothing good about it, but it seems to be the reality for more people than it should be.
What worries me is not the unrealistic notion that a single algorithm will ever replace human emotional judgment (it will not): It's the amount of frustration that blind faith in such algorithms will cause before we settle on supplementing them with common sense again.
> (...) the accuracy of the personality judgment depends on the availability and the amount of the relevant behavioral information, along with the judges’ ability to detect and use it correctly (1, 2, 5). Such conceptualization reveals a couple of major advantages that computers have over humans. First, computers have the capacity to store a tremendous amount of information, which is difficult for humans to retain and access. Second, the way computers use information—through statistical modeling—generates consistent algorithms that optimize the judgmental accuracy, whereas humans are affected by various motivational biases (27). Nevertheless, human perceptions have the advantage of being flexible and able to capture many subconscious cues unavailable to machines. Because the Big Five personality traits only represent some aspects of human personality, human judgments might still be better at describing other traits that require subtle cognition or that are less evident in digital behavior.
This last sentence is important: Looking at scores from quantitative psychological models, it is easy to forget that although a model has some predictive ability, the thought of a _complete_ model is an illusion. For example, low and high IQ scores have some merit in predicting people's success in life, but anyone who knows several people claiming to have high IQs will observe that such people's ability to actually accomplish something with their claimed ability varies greatly.
I don't doubt that psychological models can be an efficient way of filtering through many potential candidates for a position, deciding what ad to show someone or predicting the chances of someone becoming a drug addict. They could even help us see past some of our cognitive biases when making a decision. But I don't think that we will ever, at least in the next hundred years, reach a point where most people let an algorithm trump their own judgment in decisions where the emotional stakes are high. For example, in choosing a life partner, dating services will probably use such algorithms to filter the candidates, but the user will probably make the final call on who to settle with.
After all, personality tests have been around for decades, but most employers use a good old fashioned face-to-face interview to choose the person to fill the position.
> (...) the accuracy of the personality judgment depends on the availability and the amount of the relevant behavioral information, along with the judges’ ability to detect and use it correctly (1, 2, 5). Such conceptualization reveals a couple of major advantages that computers have over humans. First, computers have the capacity to store a tremendous amount of information, which is difficult for humans to retain and access. Second, the way computers use information—through statistical modeling—generates consistent algorithms that optimize the judgmental accuracy, whereas humans are affected by various motivational biases (27). Nevertheless, human perceptions have the advantage of being flexible and able to capture many subconscious cues unavailable to machines. Because the Big Five personality traits only represent some aspects of human personality, human judgments might still be better at describing other traits that require subtle cognition or that are less evident in digital behavior.
This last sentence is important: Looking at scores from quantitative psychological models, it is easy to forget that although a model has some predictive ability, the thought of a _complete_ model is an illusion. For example, low and high IQ scores have some merit in predicting people's success in life, but anyone who knows several people claiming to have high IQs will observe that such people's ability to actually accomplish something with their claimed ability varies greatly.
I don't doubt that psychological models can be an efficient way of filtering through many potential candidates for a position, deciding what ad to show someone or predicting the chances of someone becoming a drug addict. They could even help us see past some of our cognitive biases when making a decision. But I don't think that we will ever, at least in the next hundred years, reach a point where most people let an algorithm trump their own judgment in decisions where the emotional stakes are high. For example, in choosing a life partner, dating services will probably use such algorithms to filter the candidates, but the user will probably make the final call on who to settle with.
After all, personality tests have been around for decades, but most employers use a good old fashioned face-to-face interview to choose the person to fill the position.
Studies like this invariably raise the question, are computers getting very good about making judgments or are humans just piss-poor at making judgements?
My colleagues and I discussed this paper when it was published. Like lots of AI research in this area, it picks a human strawman and is sort of misleading in this regard. The human ratings are poorly designed and substandard in many ways.
It's still interesting, but there are lots of issues being swept under the rug in this area.
It's still interesting, but there are lots of issues being swept under the rug in this area.
How easy is it to game a personality judgement made by a computer vs. one by a human?
And what about those who don't indulge in the Facebook Likes game?
Hard to escape when each webpage you visit sends signals to the various social/surveillance/marketing companies. There is some legal push-back [1], but e.g. Facebook considers it 'industry standard' practice and tries to appeal/wiggle out of these data collection restrictions where it can.
[1] https://www.theguardian.com/technology/2018/feb/16/facebook-...
[1] https://www.theguardian.com/technology/2018/feb/16/facebook-...
They say their models are more accurate, but more accurate with what data? It turns out models are better than humans at predicting how people rate themselves on a personality model. So humans are rating their opinion, vs algorithms that are predicting what people would say about themselves.
So I wouldn't take those numbers at face value.
The external validity comparisons are more compelling, but still, its telling that self-reported data is better at predicting self-reported events.
It's entirely possible that computer based models are better at knowing us than humans, particularly since algorithms can pay far closer attention to us, but there's plenty of literature showing that we have the capacity for self-deception, so I'm not sure how good a ground truth you can have here.