Case Study: Accuracy & Fraud on MTurk(tumblr.gethumanoid.com)
tumblr.gethumanoid.com
Case Study: Accuracy & Fraud on MTurk
http://tumblr.gethumanoid.com/post/16940419386/accuracy-and-fraud-on-mturk-case-study
6 comments
good instructions are crucial.
I just used Humanoid for the first time. I had a job that would have taken about 5-10 hours (look up 50 ad agencies, record their Twitter account, blog URL, and their CEO's Twitter account and blog URL).
Humanoid accomplished the job for under $35. Considering I would have had to do it myself, or nag an intern for a week to do it, I consider it money well spent.
P.S. I don't know what results I would have gotten from going to Mturk directly, but the results from Humanoid were better/cheaper than my old habit of outsourcing to an Indian body shop.
(Disclosure: I am an investor. But I'm giving honest feedback.)
Humanoid accomplished the job for under $35. Considering I would have had to do it myself, or nag an intern for a week to do it, I consider it money well spent.
P.S. I don't know what results I would have gotten from going to Mturk directly, but the results from Humanoid were better/cheaper than my old habit of outsourcing to an Indian body shop.
(Disclosure: I am an investor. But I'm giving honest feedback.)
Great data, very cool how sophisticated algorithms can increase performance dramatically for human-powere tasks
What does the author mean by fraud? Do people submit junk results and hope they slip through the screening process, or it it something more insidious?
Junk results, mostly, yes. That way they can complete more taks faster and still make money because most of the time their work is not manually approved.
"Humanoid’s software identified 8% of the workforce as engaged in fraud." <--that is a big number!
Fails to mention how much they cost on top of the mturk prices to the client
Good luck to them as I think they might have made mturk easier to use for people like myself who see the value but couldn't figure out how to "make it work."