Mechanical Turk changes how we understand labor(blog.bumblebeelabs.com)
blog.bumblebeelabs.com
Mechanical Turk changes how we understand labor
http://blog.bumblebeelabs.com/mechanical-turk-changes-how-we-understand-labor
4 comments
You just inspired me for a CAPTCHA replacement. Simply asking questions can't work because we either have to have a preformulated DB of questions and answers (which could be cracked easily) or have to generate the question from a simple enough formula that a computer could derive the answer. How about, instead, we generate CAPTCHA questions on-demand from humans? Or, even better, don't bother with an automated Turing Test at all: just pay people to have a short conversation with the possible bot, and then say whether they think they're human or not.
Except MTurk and other similar services make the entire concept of CAPTCHA's obsolete with the ability to pay people to solve any human distinguishing task.
Actially, I believe this becomes a variation on the AI-Box experiment: ask the tester to not just distinguish the humanity of the challenger, but also to determine whether the challenger is a user with an honest goal to use the website, or an MT-employed opponent. The spammer can then only win if they can convince the tester to "let them out."
Interestingly, with both the challenger and tester being average humans that are badly- or un-trained on being on either side of a social engineering attack, the effectiveness of this technique will depend totally on the statistical ratio of cunning to cynicism in the average human being.
Interestingly, with both the challenger and tester being average humans that are badly- or un-trained on being on either side of a social engineering attack, the effectiveness of this technique will depend totally on the statistical ratio of cunning to cynicism in the average human being.
Except MTurk imposes a micro-cost on the attacker... until you proxy captchas into something else ("Turing porn farm").
Of course, using MTurk to generate captcha's is also a micro-cost so I don't know if you win at all (but Amazon does!).
Of course, using MTurk to generate captcha's is also a micro-cost so I don't know if you win at all (but Amazon does!).
[deleted]
[deleted]
The Calorie Tracking example (in the blog posting) illustrates that Mechanical Turk relies heavily on tremendous inequities in wealth.
A rich, fat, lazy person snaps a picture of his lavish meal with a fancy iPhone, and rather than take a brief moment to ponder the calories, … uploads it for, not just one, but three other people to ponder and estimate the calories and report back. The most dissimilar answer is rejected; that person is not compensated.
This only works if there is a very large pool of very poor people who are willing to do these mundane tasks for (fractions of) pennies. These “workers” certainly can’t afford an iPhone; they probably can’t even afford the meal they are looking at.
A rich, fat, lazy person snaps a picture of his lavish meal with a fancy iPhone, and rather than take a brief moment to ponder the calories, … uploads it for, not just one, but three other people to ponder and estimate the calories and report back. The most dissimilar answer is rejected; that person is not compensated.
This only works if there is a very large pool of very poor people who are willing to do these mundane tasks for (fractions of) pennies. These “workers” certainly can’t afford an iPhone; they probably can’t even afford the meal they are looking at.
There’s a common perception that Mechanical Turk thrives on exploiting 3rd world labor. This doesn’t appear to be the case.
From: http://behind-the-enemy-lines.blogspot.com/2008/03/mechanica... and http://asc-parc.blogspot.com/2008/07/mechanical-turk-demogra... 82% of turkers are from the US, Canada or the UK and over 75% have a Bachelor’s degree or higher.
From: http://behind-the-enemy-lines.blogspot.com/2008/03/mechanica... and http://asc-parc.blogspot.com/2008/07/mechanical-turk-demogra... 82% of turkers are from the US, Canada or the UK and over 75% have a Bachelor’s degree or higher.
You're right that "relies heavily on tremendous inequities in wealth," but not in the way you argue.
People don't spend time on MT for the money. Ok, some people do, but I believe the overwhelming motivation is the same motivation behind casual gaming.
It's just a fun thing you do because you're bored and it's like a little internet scavenger hunt.
Most Turkers are from North America, IIRC. It's more about the cognitive surplus of western laborers versus their third-world counterparts, which is related to their earning power.
Edit: Not sure why your comment was getting downvoted. I voted it back up -- it's an insightful and appropriate comment!
People don't spend time on MT for the money. Ok, some people do, but I believe the overwhelming motivation is the same motivation behind casual gaming.
It's just a fun thing you do because you're bored and it's like a little internet scavenger hunt.
Most Turkers are from North America, IIRC. It's more about the cognitive surplus of western laborers versus their third-world counterparts, which is related to their earning power.
Edit: Not sure why your comment was getting downvoted. I voted it back up -- it's an insightful and appropriate comment!
I would say its actually a case of efficiencies using batching rather than the pay discrepancy.
Taking the time out from what you're doing to write down what you ate and how many calories it was incurs a lot of "transaction" costs. Its distracting.
However, if you're on your computer and are sitting there doing many of these one right after another its a lot faster and doesn't require changing your focus.
This is the exact same reason all time management books recommend answering emails in batches.
Taking the time out from what you're doing to write down what you ate and how many calories it was incurs a lot of "transaction" costs. Its distracting.
However, if you're on your computer and are sitting there doing many of these one right after another its a lot faster and doesn't require changing your focus.
This is the exact same reason all time management books recommend answering emails in batches.
Even if it were true (and unfortunately it isn't), don't say this like it's a bad thing. Every business which relies on inequities of wealth works to eliminate them. If it succeeds, it will either adapt to a world without inequities or die.
On a similar vain I'd like to point out that Mechanical Turk jobs can only be submitted by people from the US. I can't see any reason for this. It does tend to reinforce the image of inequality.
Any thoughts and any ways to get around this?
Any thoughts and any ways to get around this?
>Mechanical Turk relies heavily on tremendous inequities in wealth.
Actually, almost all forms of employment do so as well.
Actually, almost all forms of employment do so as well.
As a developer of a calorie tracking service, also with an iPhone app in the App Store, I would point out that the example with calorie tracking does not work in RL.
1. There are hundreds of food chains in the US, with widely varying recipes, serving sizes, and nutritional contents. Even being familiar with the subject area, I generally won't be able to recognize a food. Even if I see a burger photo - how would I know the brand, flavor, whether it has cheese in it, etc. The error rate will be huge, rendering the service useless, and even worse - misleading.
2. You cannot estimate portion size on a photo, not having anything to compare with. This also results in significant errors.
I am not arguing against other uses of MT, just pointing out that this is a not very well thought through example.
MT is applicable only for very rudimentary tasks, requiring absolutely zero qualifications and training. There are fewer such tasks around than it looks at the first glance, as this example demonstrates.
1. There are hundreds of food chains in the US, with widely varying recipes, serving sizes, and nutritional contents. Even being familiar with the subject area, I generally won't be able to recognize a food. Even if I see a burger photo - how would I know the brand, flavor, whether it has cheese in it, etc. The error rate will be huge, rendering the service useless, and even worse - misleading.
2. You cannot estimate portion size on a photo, not having anything to compare with. This also results in significant errors.
I am not arguing against other uses of MT, just pointing out that this is a not very well thought through example.
MT is applicable only for very rudimentary tasks, requiring absolutely zero qualifications and training. There are fewer such tasks around than it looks at the first glance, as this example demonstrates.
This story makes me really sad. You know why? Because it means that humans are now so common and cheap that we can use them for menial labor.
People are now cheaper than machines :(
This is a reversal of 2000 years of progress.
People are now cheaper than machines :(
This is a reversal of 2000 years of progress.
Mechanical Turk is generally used for things that machines still can't do with nearly the accuracy that humans can. Image recognition, facial recognition, translation, dictation, etc.
No matter how much hardware you throw at those problems, you can't (yet) match what a human brain can do in a split second. I don't think that's depressing.
No matter how much hardware you throw at those problems, you can't (yet) match what a human brain can do in a split second. I don't think that's depressing.
Cheer up. People are not cheaper than machines, they're just better at some tasks. You'd have to complete tens of thousands of those stupid little tasks to pay your rent for a month...if they could be done by machine reliably, they would be.
So I built a very simple add on to our timeclock (which is essentially an ELO touch screen and a PC) and had it use a web cam to take a picture of each person punching. Then we created a program that showed turk users a person's badge picture and then the picture taken at the punch and asked "Is this the same person?" (with answers of Yes/No/Maybe) Doing that virtually eliminated the problem overnight.
I agree with the author that M.T. won't completely change the market. But what it will do is allow people to monitize their spare seconds and in doing so make a lot of the "no-brainer" grunt work that companies have to do a lot cheaper