Poll: What experiment to run on AWS mechanical turk?
13 comments
I'd love to see a prisoner's dilemma experiment via MTurk. These tend to be financial in nature, though; can you set variable payment rates per task, depending on external factors?
you can pay out arbitrary bonuses once they've answered.
so you could easily do single-round prisoner's dilemma, and have amazon pair up 2 (different) folks at a time, and when you get the results back, look to see what the folks in each pair did, and award bonuses appropriately.
i'm not sure if there is a way to do an iterated version.
so you could easily do single-round prisoner's dilemma, and have amazon pair up 2 (different) folks at a time, and when you get the results back, look to see what the folks in each pair did, and award bonuses appropriately.
i'm not sure if there is a way to do an iterated version.
A friend last week told me about a business he's working with. They do Natural Language Processing to scan thousands of articles about a specific topic/product to see if they are positive, negative or neutral. They are then hoping to turn around and sell the business intelligence.
I was wondering how it would compare on a cost/effectiveness basis with the Turk.
Let us know what your experiment is and how you fared.
I was wondering how it would compare on a cost/effectiveness basis with the Turk.
Let us know what your experiment is and how you fared.
See if you can use MT to provide the fitness function for some genetic algorithm, evaluate the results, tune it, and profit!
Some months ago I tried to set up a project to transcribe Alexander Shulgin's Lab books but Mturk wasn't available outside US.
http://www.erowid.org/library/books_online/shulgin_labbooks/
http://www.erowid.org/library/books_online/shulgin_labbooks/
Go meta - use Mechanical Turk to ask what would be a really cool idea to run using Mechanical Turk.
How about creating an experiment and designing it right on turk? You create a rough sketch and then you auto-correct it using people?
You mean keep tinkering with the study until it "works"?
If you're not careful, you could get http://en.wikipedia.org/wiki/Publication_bias on a massive new scale...
It would be interesting to try to find scientifically optimal ways of doing study iteration, or approaches to interpreting the complete (experiment_rev, results)-list from another researcher...
If you're not careful, you could get http://en.wikipedia.org/wiki/Publication_bias on a massive new scale...
It would be interesting to try to find scientifically optimal ways of doing study iteration, or approaches to interpreting the complete (experiment_rev, results)-list from another researcher...
I'd pay $.05 to get the them to up vote my site for certain search terms on Google's new "Search Wiki". See what happens.
What was your previous experience?
we used the Turk service to basically do OCR on about 3k images - http://shout.setfive.com/2008/11/20/artificial-artificial-in...
We have some downtime right now and we are narrowing down the experiment between the following:
1. Some sort of cooperative art project - inspired by http://www.tenthousandcents.com 2. A massive chess game that forks every time a move is made (probably a poor idea) 3. Some implementation of the Prisoner's dilemma game. 4. Any time of game to get a sense of how risk seeking or risk averse the Turks are - possibly a tie in with the Monty Hall Problem.
Most of these ideas are centered around game theory but we are pretty much open to anything (within reason). Any votes or opinions on what would be the most interesting/entertaining?
-Thanks