A Plea to Amazon: Fix Mechanical Turk(behind-the-enemy-lines.blogspot.com)
behind-the-enemy-lines.blogspot.com
A Plea to Amazon: Fix Mechanical Turk
http://behind-the-enemy-lines.blogspot.com/2010/10/plea-to-amazon-fix-mechanical-turk.html
11 comments
I have a product that I hope to launch before Jan 1 that will address these issues.
Any details on this at all? A hint?
I like Panos have been doing this for more than 4 years, and I've seen no silver bullets. I'd like to think we're (CastingWords) doing it better than anyone else, and I don't even know what a silver bullet would look like at this point. Though I have to say that Panos's list is a good start.
I like Panos have been doing this for more than 4 years, and I've seen no silver bullets. I'd like to think we're (CastingWords) doing it better than anyone else, and I don't even know what a silver bullet would look like at this point. Though I have to say that Panos's list is a good start.
Is this something that runs on top of MTurk or similar existing crowd-sourcing networks or a new crowd-sourcing network?
Tell us where to sign up to be notified of beta.
Please sign up for the beta here:
https://spreadsheets.google.com/viewform?formkey=dDNkei1Hb0l...
https://spreadsheets.google.com/viewform?formkey=dDNkei1Hb0l...
I have an API that addresses some of these issues in alpha right now: http://houdinihq.com. Feel free to get in touch if you'd like - my email's in my profile.
I have used Mechanical Turk quite extensively the last few months. But I have always used it via Crowdflower. They offer a more sensible API and interface. And they keep scores on all workers, and calculate how well they perform. This influcences the weight of their job submissions. It takes a bit getting used to their terms, but I'm quite happy with the end results.
CrowdFlower has some annoying particularities which are undocumented.
In particular:
The CSV parser doesn't work / is too strict / something. The only way to upload complicated data files in as "JSON". The only problem is they don't accept standard JSON. You cannot have a comma separating the rows. (!?) I have no idea how much time I spent trying to figure this out.
Additionally, if your task is too hard, and more than 50% of judgments are rejected, they auto-pause your task. When you resume it, it will simply pause a few minutes later. So, for certain tasks, you simply cannot finish the job. Their rationale is that CrowdFlower pays for the judgments that are discarded, and you pay for only the judgments they keep, so if they discard too many judgments they lose money. I would rather just pay extra money for tasks that are very hard.
In particular:
The CSV parser doesn't work / is too strict / something. The only way to upload complicated data files in as "JSON". The only problem is they don't accept standard JSON. You cannot have a comma separating the rows. (!?) I have no idea how much time I spent trying to figure this out.
Additionally, if your task is too hard, and more than 50% of judgments are rejected, they auto-pause your task. When you resume it, it will simply pause a few minutes later. So, for certain tasks, you simply cannot finish the job. Their rationale is that CrowdFlower pays for the judgments that are discarded, and you pay for only the judgments they keep, so if they discard too many judgments they lose money. I would rather just pay extra money for tasks that are very hard.
Crowdflower's API is excellent in comparison. And the UI is much better to boot.
We find it restricts the kinds of work we like to do too much - but it also restricts it to the kinds of work that Crowdflower can reasonably return good results on. So that seems like a good trade off.
I recommend Crowdflower as a starting place to anyone thinking of using MTurk.
We find it restricts the kinds of work we like to do too much - but it also restricts it to the kinds of work that Crowdflower can reasonably return good results on. So that seems like a good trade off.
I recommend Crowdflower as a starting place to anyone thinking of using MTurk.
Fix these problems and you may have yourself a viable startup.
These change requests are (mostly) from one particular perspective, so it ignores some other problems from the worker's perspective; namely, that the market hasn't quite settled on reasonable rates for work, and that a worker has insufficient protection from exploitive work providers. These are also serious problems that could hinder it as a platform.
I think the relative supply and demand are going to determine prices here, just like any other market. A price floor could be imposed, but this would likely price some work out of the market or lead to other work-arounds as trading partners try to avoid constraints.
As far as protecting workers, in a survey I ran last year, there wasn't any evidence that workers on MTurk feel more exploited by their online bosses compared to their offline bosses:
http://www.technologyreview.com/blog/arxiv/24646/
Though obviously those that feel particularly mistreated could be selecting out of the market.
As far as protecting workers, in a survey I ran last year, there wasn't any evidence that workers on MTurk feel more exploited by their online bosses compared to their offline bosses:
http://www.technologyreview.com/blog/arxiv/24646/
Though obviously those that feel particularly mistreated could be selecting out of the market.
Take a look at this article: http://behind-the-enemy-lines.blogspot.com/2010/07/mechanica...
You will see that the low wages are a direct result of a lack of a good reputation system
You will see that the low wages are a direct result of a lack of a good reputation system
Completely agree with his sentiment regarding TurkIt and the prospect of making iterative tasks easily accessible to the masses. Read some of the papers that Greg Little has put out from MIT to open up your eyes to some advanced uses of Mechanical Turk.
Using the RTurk gem and my recently released Turkee gem ( http://github.com/aantix/turkee ), I've been able to quickly throw together a demo Rails application that allows Turkers to vote on previous Turker submissions or use other turker's submissions as the basis of their own own submission.
With my experience under this type of interface most Turkers will vote for one of the previous submissions but if you offer a small bonus for the highest voted submission, more turkers will take a chance at submitting something original (even if it's a trivial 50 cents bonus).
The demo's code is available under my github account (aantix/Turkee-Iterator). Here's some of screenshots of my demo; the top image shows my initial parameters for setting up my Turkee task. The second screen shows the responses I received from the Turkers. The third screen shows the form that was presented to the Turkers (either vote or submit a joke).
http://imgur.com/13rCM.png
http://imgur.com/ghveH.png
http://imgur.com/229uc.png
Right now I am writing a game that generates content using a couple iterative approaches to content generation with Mechanical Turk (hence my interest in this area).
Don't hesitate to drop me a line if you have any problems with getting my demo running.
Using the RTurk gem and my recently released Turkee gem ( http://github.com/aantix/turkee ), I've been able to quickly throw together a demo Rails application that allows Turkers to vote on previous Turker submissions or use other turker's submissions as the basis of their own own submission.
With my experience under this type of interface most Turkers will vote for one of the previous submissions but if you offer a small bonus for the highest voted submission, more turkers will take a chance at submitting something original (even if it's a trivial 50 cents bonus).
The demo's code is available under my github account (aantix/Turkee-Iterator). Here's some of screenshots of my demo; the top image shows my initial parameters for setting up my Turkee task. The second screen shows the responses I received from the Turkers. The third screen shows the form that was presented to the Turkers (either vote or submit a joke).
http://imgur.com/13rCM.png
http://imgur.com/ghveH.png
http://imgur.com/229uc.png
Right now I am writing a game that generates content using a couple iterative approaches to content generation with Mechanical Turk (hence my interest in this area).
Don't hesitate to drop me a line if you have any problems with getting my demo running.
My main issue with Mturk is not being able to sign up as a non-US company.
It's not that I'm angry at Amazon or anything, I just wish that they would come to the conclusion that our money is just as green as everyone elses.
It's not that I'm angry at Amazon or anything, I just wish that they would come to the conclusion that our money is just as green as everyone elses.
Same with me. I spent over 1k with a friend on turk, even reported blatant security issues to one of the PM's yet I was unable to sign up and had to get my American friend to do it.
I see no reason why they can't open it up to Canada. Terribly frustrating. Take my money!
I see no reason why they can't open it up to Canada. Terribly frustrating. Take my money!
Aw dammit, Turk is great.
I spent $300 in one day on Turk and Amazon gave me a personal account manager... And that's the problem, if you spend $300 on Turk in a day you're a high roller, Turk just isn't that big of a market. I've had days when I've been responsible for 20% of the HITs in the system and I'm just a poor computer programmer who writes code for a penny a line. I just got an email the other day from a Turk who was begging for more HITs.
Personally I've got no problem with the API and for certain kinds of tasks I've got good answers for the quality problems. For other ones, I haven't figured it out yet.
If you think Amazon's offering a service that's too bare bones, you ought to start a start-up that layers services on top of it. It may be a tough row to hoe though... My layer that interfaces to turk is specialized to my purposes and isn't spiffy, commercial and easy to use. It would probably cost you 20-50 times as much to develop something "usable" than it cost me to develop something that fits my workflow. Unless you can amortize this cost over a LARGE number of customers, your spiffy new system is going to cost more than it costs to pay the Turks. And good luck getting Angel or VC money to develop something that might never get all that big... And that will probably get regulated with the government when they find out that you're getting people to work for you for $2.50 an hour!
For now I'm tickled pink because I can get labor cheaper than my competitors can -- I can escape the economic problems that keep silicon valley companies going in circles chasing each other's tails.
What I really wish is that I could pay people Facebook credits to do work for me: my guess is that I could be paying people $1.25 an hour that way... However, Facebook knows the first thing people would do if it was easy to pay credits out to people is create gambling apps, so they don't make it easy.
I spent $300 in one day on Turk and Amazon gave me a personal account manager... And that's the problem, if you spend $300 on Turk in a day you're a high roller, Turk just isn't that big of a market. I've had days when I've been responsible for 20% of the HITs in the system and I'm just a poor computer programmer who writes code for a penny a line. I just got an email the other day from a Turk who was begging for more HITs.
Personally I've got no problem with the API and for certain kinds of tasks I've got good answers for the quality problems. For other ones, I haven't figured it out yet.
If you think Amazon's offering a service that's too bare bones, you ought to start a start-up that layers services on top of it. It may be a tough row to hoe though... My layer that interfaces to turk is specialized to my purposes and isn't spiffy, commercial and easy to use. It would probably cost you 20-50 times as much to develop something "usable" than it cost me to develop something that fits my workflow. Unless you can amortize this cost over a LARGE number of customers, your spiffy new system is going to cost more than it costs to pay the Turks. And good luck getting Angel or VC money to develop something that might never get all that big... And that will probably get regulated with the government when they find out that you're getting people to work for you for $2.50 an hour!
For now I'm tickled pink because I can get labor cheaper than my competitors can -- I can escape the economic problems that keep silicon valley companies going in circles chasing each other's tails.
What I really wish is that I could pay people Facebook credits to do work for me: my guess is that I could be paying people $1.25 an hour that way... However, Facebook knows the first thing people would do if it was easy to pay credits out to people is create gambling apps, so they don't make it easy.
Chris Conley, a member of this community and a well known Philly developer, has been building a powerful API for developers to use to interact with Mechanical Turk (it started with his own frustrations using theirs). You can check it out here: http://houdinihq.com/.
If you're looking for a solution to some of the problems addressed in this article, his API (and the corresponding Rails gem) are a good place to start.
If you're looking for a solution to some of the problems addressed in this article, his API (and the corresponding Rails gem) are a good place to start.
Great community and service, terrible and confusing interface. I wonder maybe their UI is so bad as they only want to attract a certain demographic, though it seems their demographic(turkers) runs beyond just computer nerds.
I think MT is failing because they only allow busiless in the US, which is a tiny market for jobs that pay pennies/hour. But if Amazon managed to open up say, India or China, there would be a much richer ecosystem in place.
An interesting finding is that it costs about $5K-10K to open a US bank account from overseas, post 9/11. Pre 9/11 it was probably around $100.
Although you can use MT credit to buy stuff out of the Amazon store.
An interesting finding is that it costs about $5K-10K to open a US bank account from overseas, post 9/11. Pre 9/11 it was probably around $100.
Although you can use MT credit to buy stuff out of the Amazon store.
Has anyone tried CloudCrowd? They have a similar system to MTurk but it's built on facebook and it's a managed process similar to crowdflower... Interested to hear any experiences.