10,000 Mechanical Turkers work separately to draw a 100 dollar bill.(tenthousandcents.com)
tenthousandcents.com
10,000 Mechanical Turkers work separately to draw a 100 dollar bill.
http://www.tenthousandcents.com/index.html
49 comments
If you scroll down you can see that some people drew completely unrelated things. One guy doodled a stick figure. Another wrote "$0.01!! REALLY?!" The last guy did the drawing, then wrote "i hate u" over top.
Paying $0.02 instead of $0.01 for a hit gets you faster response and better quality usually
That kind of stuff is pretty rare. Turkers are pretty hard-working/honest people on the whole I guess out of 10,000 maybe 10-20 screwed around. That's less than .2%.
They're certainly harder working than I'd be, given the amount of effort for the return.
Obviously the dollar bill guys picked the worst ones because they wanted to make their point about crowdsourcing. Whatever that point might be.
In any case, it's more interesting to stare at than just 8 well-done samples.
In any case, it's more interesting to stare at than just 8 well-done samples.
Looking at the bill, the number of bad cells seems far higher than you suggest. Some people didn't absolutely screw around but many did not finish the work. The percentage gets pretty bad in some of the high detail areas.
I'm a huge fan of MT, and do think that it can be used to get pretty accurate/high quality results. You can actually have multiple people work on each HIT and then cross-check or chose the best
I actually posted about our experience with MY yesterday here: http://blog.newscred.com/?p=157
I actually posted about our experience with MY yesterday here: http://blog.newscred.com/?p=157
An algorithm you could use: try three times, prefer the most similar two ("map"), look in those for puzzle-piece matches with the nearest neighbors ("reduce").
http://www.tenthousandcents.com/gallery.html
This is the most interesting piece. You are seriously missing something if you are not checking this out. There are multiple sections -
Detail (Mind blowing! - people putting in so much into this)
Creativity - All the funny stuff
Random art - hearts, texts and all that.
This is the most interesting piece. You are seriously missing something if you are not checking this out. There are multiple sections -
Detail (Mind blowing! - people putting in so much into this)
Creativity - All the funny stuff
Random art - hearts, texts and all that.
I think its funny that folks feel this stuff is so baffling. It think what's causing this alarming "HEY" in people is the price. Understandable really, who the hell wants to do anything for .02 cents. Yet, again, we got blinders on. That's not .02 cents an hour, time really isn't defined, either is real effort, or care and quality of effort.
We just jump to the obvious answer in our heads without thinking about why, why WOULD someone do this. Why do people participate in social media? Why does the average teen check their myspace page 17 times a day? Who's getting paid to make 2.0 such an interesting place, well we sure aren't. Yet in turk land, the name of the game isn't to wake up and get excited about doing a .02 cent task. Its how fast can I do that so it doesn't feel like I did a .02 cent task. I'm bored, watching tv, why not make a few cents. Crazy, I know, yet, again, hello people are doing it.
Then we jump into the next assumption, well most of these folks have to be indian or chinese. Gold farmers sleeping in cots, that's it, thats gotta be it. Feel better? Wrong, 80% of these turks are not chinese gold farmers. They average joes here in the states buying milk and bacon just like you do. SHOCKING!
The sad thing is we hone in on all the .02 cent stories in turk, and yes there are plenty of them. But most turks I believe endure that cent crap to get to bigger fish. The "it took me less than 30 seconds for a buck" HITS, and I did 50 of those today. Now the money isn't so bad.
For people requesting the work, paying some 2 cents is a great way to get crap data, good luck with that. Course at a large volume of stuff, maybe you don't care, but in general turks don't do 2 cent tasks for 2 cents, they do it cause they are bored, multi-tasking, its interesting or fun- drawing 100 bill rather interesting, or cause its fast as hell to do.
We just jump to the obvious answer in our heads without thinking about why, why WOULD someone do this. Why do people participate in social media? Why does the average teen check their myspace page 17 times a day? Who's getting paid to make 2.0 such an interesting place, well we sure aren't. Yet in turk land, the name of the game isn't to wake up and get excited about doing a .02 cent task. Its how fast can I do that so it doesn't feel like I did a .02 cent task. I'm bored, watching tv, why not make a few cents. Crazy, I know, yet, again, hello people are doing it.
Then we jump into the next assumption, well most of these folks have to be indian or chinese. Gold farmers sleeping in cots, that's it, thats gotta be it. Feel better? Wrong, 80% of these turks are not chinese gold farmers. They average joes here in the states buying milk and bacon just like you do. SHOCKING!
The sad thing is we hone in on all the .02 cent stories in turk, and yes there are plenty of them. But most turks I believe endure that cent crap to get to bigger fish. The "it took me less than 30 seconds for a buck" HITS, and I did 50 of those today. Now the money isn't so bad.
For people requesting the work, paying some 2 cents is a great way to get crap data, good luck with that. Course at a large volume of stuff, maybe you don't care, but in general turks don't do 2 cent tasks for 2 cents, they do it cause they are bored, multi-tasking, its interesting or fun- drawing 100 bill rather interesting, or cause its fast as hell to do.
What kind of people think of stuff like this?
Or more precisely - what kind of people think of stuff like this, sober up, and then actually implement it?
Or more precisely - what kind of people think of stuff like this, sober up, and then actually implement it?
People who eventually end up doing something meaningful, because they're not afraid to try.
"Afraid" is not even close to the right word for this context.
When I consider something like this, I don't think: "Oh, that's scary and/or challenging!" Instead, I think: "given all other possible uses of my time, is this one the best use possible?" Usually, the answer is no.
For me, if there's any "meaning" in this project, it's in divining why someone would choose to begin (or participate in) this in the first place.
When I consider something like this, I don't think: "Oh, that's scary and/or challenging!" Instead, I think: "given all other possible uses of my time, is this one the best use possible?" Usually, the answer is no.
For me, if there's any "meaning" in this project, it's in divining why someone would choose to begin (or participate in) this in the first place.
If the question is "given all other possible uses of my time, is this one the best use possible?", then the answer will be "no" for almost any given value of "this". A likely result is doing nothing while one waits for the "perfect" thing to do - it might not apply as much to people on news.yc, but this does describe many people I know.
If the question is "given all the ideas I have right now, and considering the effort to implement each, should I work on this?", the answer might be different.
If the question is "given all the ideas I have right now, and considering the effort to implement each, should I work on this?", the answer might be different.
In fact, it is impossible to determine the best possible use of your time without determining, without doubt, the meaning of life.
Did you ask yourself that before posting that comment?
People have different interpretations of what they consider meaningful.
People have different interpretations of what they consider meaningful.
Actually, I did ask myself that question (clearly, I'm not always listening to my own answers).
Re-reading what I wrote, it sounds harsher than I had intended. Here's what I was trying to get at: for me, most of the meaning in projects like this (IMO) lies in the interpretation of why the artist chose to do it at all. My own limit is not typically fear, but indecisiveness.
Re-reading what I wrote, it sounds harsher than I had intended. Here's what I was trying to get at: for me, most of the meaning in projects like this (IMO) lies in the interpretation of why the artist chose to do it at all. My own limit is not typically fear, but indecisiveness.
Read 'Round Ireland with a Fridge'.
http://www.amazon.com/gp/reader/0312274920/ref=sib_dp_pt#rea...
[deleted]
Artists.
What really baffles me is the number of people that start out with the foreground and then destroy most of their work by slowly filling in the background. No wonder that it took them 5 month.
Great project though and a nice demonstration of what can be done with mechanical turk.
With a single print ($100) they recovered the costs of the entire labor ($100, or 10,000 cents). This is a great metaphor of our modern production chain, isn't it? (i. e.: How production exploits cheap labor)
What metaphor would we use to explain the guy who did this on our production line?
http://imagesocket.com/view/lol100b69.PNG
http://imagesocket.com/view/lol100b69.PNG
I am assuming that there where no QA measures. Interesting to see the level of quality they got without it.
I'm pretty sure the average Mechanical Turk user could have scraped together $100, if they'd thought of it.
If you're going to talk about Big Capitalism exploiting the cowering, puny forces of labor, you might want to pick a business where the startup cost is in the low three figures.
If you're going to talk about Big Capitalism exploiting the cowering, puny forces of labor, you might want to pick a business where the startup cost is in the low three figures.
Isn't it illegal to sell life-sized colour prints of a $100 bill? Quality aside...
This is only the front. You check the backs of your US bills, don't you?
One-sidedness is one of the legal requirements, but it's not the only one.
http://www.ustreas.gov/usss/money_illustrations.shtml
http://www.ustreas.gov/usss/money_illustrations.shtml
I've always been a little confused... Who's time is worth so little that they'd actually see a single cent worth doing one of these drawings? Kids? Third world laborers?
I can see kids doing this, though I doubt that many would have access to the accounts and bank information necessary. As per third worlders: if a person's situation were such that a cent per minute was a reasonable wage, how could they afford a computer and an Internet connection? I'm imagining vast Chinese sweatshops with workers making .5 on the cent.
Is my analysis wrong here?
I can see kids doing this, though I doubt that many would have access to the accounts and bank information necessary. As per third worlders: if a person's situation were such that a cent per minute was a reasonable wage, how could they afford a computer and an Internet connection? I'm imagining vast Chinese sweatshops with workers making .5 on the cent.
Is my analysis wrong here?
A classic and popular misconception that "no one in their right mind would do this", yet they do. The time for effort formula trips up a lot of people, your not the first one come up with this analysis.
The problem you have is the lens you're looking through. No one paid you to post here, no one paid you to participate in a social network, watch a youtube video, let alone make a blog, yet a whole heck of alot of the folks on the planet do it.
People need to stop looking at turk through the "job" lens because it will never ever make sense. But if you look at it through the multitasking, social media, bored outa my gord, what the hell, yearning to participate in something, be apart of something, advisory and authority lens it makes much more sense.
Meanwhile, read up on turks, get to know their behaviors and find out the why behind their actions.
http://www.floozyspeak.com/blog/archives/2008/08/valley_of_t...
Don't take my word for it, go ask them yourself, make HIT, put it up there and see for yourself. But take the traditional "this is a job" lens off cause that is a classic misguided view of mechanical turk.
The problem you have is the lens you're looking through. No one paid you to post here, no one paid you to participate in a social network, watch a youtube video, let alone make a blog, yet a whole heck of alot of the folks on the planet do it.
People need to stop looking at turk through the "job" lens because it will never ever make sense. But if you look at it through the multitasking, social media, bored outa my gord, what the hell, yearning to participate in something, be apart of something, advisory and authority lens it makes much more sense.
Meanwhile, read up on turks, get to know their behaviors and find out the why behind their actions.
http://www.floozyspeak.com/blog/archives/2008/08/valley_of_t...
Don't take my word for it, go ask them yourself, make HIT, put it up there and see for yourself. But take the traditional "this is a job" lens off cause that is a classic misguided view of mechanical turk.
I think you overestimate the value of most of the worlds workers.
Let's say revenue is 1cent an minute for 40 hours a week and 48 weeks a year. That's 9,216$ / year. Split the cost of a low cost energy efficient PC 300$, electricity ~100$, and internet 400$/year, over 1 year. And your left with ~8416$ which works out to ~700$ a month. In 2004 a southern China factory job would pay about $120 a month for long hours and often dangerous conditions. There are more than 4 billion people who gladly take that job but most of them don't have the skills to do so.
PS: If you think my numbers are a little off 8000$/year places someone in the top ~15% of the worlds income earners. (http://www.globalrichlist.com/)
Let's say revenue is 1cent an minute for 40 hours a week and 48 weeks a year. That's 9,216$ / year. Split the cost of a low cost energy efficient PC 300$, electricity ~100$, and internet 400$/year, over 1 year. And your left with ~8416$ which works out to ~700$ a month. In 2004 a southern China factory job would pay about $120 a month for long hours and often dangerous conditions. There are more than 4 billion people who gladly take that job but most of them don't have the skills to do so.
PS: If you think my numbers are a little off 8000$/year places someone in the top ~15% of the worlds income earners. (http://www.globalrichlist.com/)
Ops, I knew that number looked to large. I multiplied by an extra 8. Anyway, lot's of people live off of less than 1cent an min of work, but you would need to significantly cut down the PC and internet cost's to make it work out. Anyway, it can still work for a surprising number of people as 1k/year still put's you into the top 1/2 of the worlds workers.
A person that makes 1k/year will hardly be able to buy a PC.
That person will probably not know how to use a computer, read or understand English.
That person will probably not know how to use a computer, read or understand English.
True, but I think this is what the 1 laptop per child concept is all about. US minimum wage is 6.25/hour in other economies over a billion people would be happy with 1-2% of that. The trick is finding something useful for them to do and handing them the tools to do so. Anyway, I expect the cost of outsourcing low end jobs is only going to go down as digital work becomes more common.
PS: My point is mostly about what constitutes a "living wage" in most of the world and has little to do with Amazon Mechanical Turk as it exists right now. I expect most people using Turk are making a lot more than 1cent / min of work or are just doing it as a hobby where the cash value is almost zero incentive.
PS: My point is mostly about what constitutes a "living wage" in most of the world and has little to do with Amazon Mechanical Turk as it exists right now. I expect most people using Turk are making a lot more than 1cent / min of work or are just doing it as a hobby where the cash value is almost zero incentive.
Hrm. Still seems weird to me. I post here because it's pretty explicit that I do it for my own reasons. If I were to make 2 cents per post, that would make it somehow demeaning for me, I think.
It does appear to work though, so maybe that's not a valid concern for many people. I wonder how exhaustible a resource people like that are, though. If MT were to become more widely used, would it fall off a cliff as the supply of people doing it "just for the heck of it" dried up and they are forced to ask for market rates?
It does appear to work though, so maybe that's not a valid concern for many people. I wonder how exhaustible a resource people like that are, though. If MT were to become more widely used, would it fall off a cliff as the supply of people doing it "just for the heck of it" dried up and they are forced to ask for market rates?
Things do seem more boring when you're making money for it. Does this effect have a name?
I'm reminded of the first time I ever went to a casino. After an hour of exploring and putting the occasional dollar in a slot machine, I was about five dollars down. "Oh well, I thought, five dollars for an hour's entertainment is pretty good value!". Then on the next machine, I won ten bucks, and found myself thinking "An hour's work for five dollars? What a gip!"
I'm reminded of the first time I ever went to a casino. After an hour of exploring and putting the occasional dollar in a slot machine, I was about five dollars down. "Oh well, I thought, five dollars for an hour's entertainment is pretty good value!". Then on the next machine, I won ten bucks, and found myself thinking "An hour's work for five dollars? What a gip!"
workers remorse.
Please don't patronize me. I never said, "no one in their right mind would do this." I asked, "why would someone do this if the primary motivation was money?" If the primary motivation isn't money, and people are looking to alleviate boredom, all you needed to do was say that. understand that the reason I asked was because I tried it once and couldn't stand it.
That said, your study does answer my question. People see it as easy money that they can pursuit while doing other things or when there is nothing else to do, which makes sense.
That said, your study does answer my question. People see it as easy money that they can pursuit while doing other things or when there is nothing else to do, which makes sense.
> Who's time is worth so little that they'd actually see a single cent worth doing one of these drawings? Kids? Third world laborers?
Unemployed Java programmers?:)
Unemployed Java programmers?:)
I guess for most people it's not the money, but the feeling of accomplishment and the sense of community - teaming up to achieve a bigger goal. I doubt that sweatshops would do this - they are busy breaking captchas and mining gold in WoW.
Except, how do you know what the bigger goals are? I signed up for MT a while ago thinking I should give it a shot to see what it was like. I had no idea what the tasks added up to on a grander scale and I found that it really wasn't worth my time.
Edit: Are you sure that folks don't use MT to break captchas?
Edit: Are you sure that folks don't use MT to break captchas?
a salon.com article about the motivation of turkers has been posted here not to long ago:
http://www.salon.com/tech/feature/2006/07/24/turks/
It's certainly nothing for me, but it somehow attracts a lot of people. Is there something about breaking captchas in their terms of service?
It's certainly nothing for me, but it somehow attracts a lot of people. Is there something about breaking captchas in their terms of service?
FTA:
Efficient Frontier, a search engine marketing firm, has used Mturk.com to accomplish tens of thousands of tasks since early 2006. Efficient Frontier helps companies figure out which keywords will bring Web surfers to their sites. With Mturk.com, Efficient Frontier can afford to pay three different people to look at each potential keyword, and vote whether those words are relevant to a given site. It costs the company just 4.5 cents to test each keyword, paying 1.5 cents to Amazon, and 1 cent each to three turkers.
That is sheer genius.
Efficient Frontier, a search engine marketing firm, has used Mturk.com to accomplish tens of thousands of tasks since early 2006. Efficient Frontier helps companies figure out which keywords will bring Web surfers to their sites. With Mturk.com, Efficient Frontier can afford to pay three different people to look at each potential keyword, and vote whether those words are relevant to a given site. It costs the company just 4.5 cents to test each keyword, paying 1.5 cents to Amazon, and 1 cent each to three turkers.
That is sheer genius.
this is the same idea as the 10,000 sheep art project, done by the same guy, aaron koblin
whee
whee
This isn't exactly new. It was also one of the first projects like this done with Turk (ie, an art project and not just menial data entry).
Guess the Mechanical Turk isn't good for, say, important data entry (without a system to set up cross-checks).
Pretty cool, though.