Mechanical Turk: Now with 40.92% Spam.(behind-the-enemy-lines.blogspot.com)

behind-the-enemy-lines.blogspot.com

http://behind-the-enemy-lines.blogspot.com/2010/12/mechanical-turk-now-with-4092-spam.html

27 comments

At first, I thought MTurk was good for getting translation work done, and checked by others. It worked OK, but I had to be clever to avoid the cheaters who would ignore instructions not to use machine translation. After refining my tactics, talking to a lot of workers, and fending off the crooks, I got a few decent translations (for my iPhone app).

Then I tried a new tactic. I hired some pros from various translation websites. All were more expensive (money-wise) than MT, but the results were better, and I spent a lot less time. Some were even close to MTurk prices.

So, I don't use MTurk anymore. Maybe if I had a more mindless task I'd try again, but it is a lot of hassle.

I know this isn't exactly what the author means by "spam" - but even the workers on that site are often dishonest. It's a total black market.

j2d2j2d216 anni fa

I had good success stopping cheating by using images of text instead of actual text. Typing in a foreign language is not trivial if it's not something you're used to doing.

I used a python wrapper for pango-view to generate the text. I have a python module for this here: https://github.com/j2labs/pyango_view

Nothing complicated about what I'm doing but the github page shows how nice the text can really look.

j2d2j2d216 anni fa

It might also be of interest that I recently posted a gist that shows how to post some basic mechanical turk tasks using Python's boto library.

https://gist.github.com/740267

trotsky16 anni fa

It seems to be a failed social experiment to me, there is definitely a negative feedback loop there. Bad HITS, low wages, withheld payments, bad workers, sweatshops. Hard to imagine it will get any better.

I wonder if you reversed the process and had projects bid on workers if it could counter the race to the bottom effect. Include a system where work could be divided into common types and include worker metrics and feedback that captured any specialization. Perhaps allow some workers to fill a managerial role, where they acquire large blocks and have a trusted group they farm out work to. Presumably wages would go up but bad work would go down, perhaps offsetting it somewhat.

byoung216 anni fa

It sounds like you have a great idea for a startup to bring some competition to this area.

eru16 anni fa

Mturk does have `qualification' (or so) that do capture specialization. Your ideas seem interesting, though.

d_r16 anni fa

Not 100% relevant to this thread, but if it helps, I've had success with http://www.icanlocalize.com for localizing apps. They assign a human translator for your strings and are accommodating with custom directions you might have. They'll even take your strings file from Xcode.

(Disclaimer: I'm not affiliated with this site, your mileage may wary.)

DenisM16 anni fa

what's more important - they will return you files fit for XCode! icanlocalize is awesome.

rgrieselhuber16 anni fa

I'm starting to use MyGengo quite a bit for translations. It's more expensive than MT but the quality is great and service is super-fast.

gcv16 anni fa

Fascinating.

My experience with MTurk suggests that Amazon has mostly abandoned the product. Amazon still uses it for product classification, but does not participate in the community forums (even on the official AWS MTurk developer forum). No new features have appeared in several years, and enhancement requests have fallen on deaf ears.

That said, MTurk still works. The API still does what its documentation claims. People still perform tasks. Payment still works.

nmcfarl16 anni fa

I have to say that although it feels that way, lately they've been sort of active.

They've been using twitter since October at: http://twitter.com/amazonmturk

And they just launched launched a blog at: http://mechanicalturk.typepad.com/

Largely I attribute this to Natala Menezes, their newish Product Manager.

natala16 anni fa

Nathan, thank you for the kind words -- but I really can't take any credit. Just know we are actively listening and we take all the feedback, even the controversial bits, very seriously.

khafra16 anni fa

The obvious question: do you have a computer program set to alert you if "natala" appears on a tech site in a mturk context, or a mechanical turk task set with a 10 cent payment to email you whenever you're mentioned?

natala16 anni fa

;-) No. I found out the old fashioned way -- a friend saw the post and sent it to me. He reads YCombinator news much more frequently than me! It does help to have a unique name...

zacharycohn16 anni fa

Google Alerts.

lsb16 anni fa

MTurk has been in beta for a while. What's new about MTurk since last year?

keith_rich16 anni fa

But what have they done lately? Do you seriously consider opening a blog and a twitter account to be "sort of active"? The blog contains a single fluffy marketing piece.

staunch16 anni fa

It's unbelievable that no one has swooped in to crush MTurk given how little Amazon has done to develop it since 2005(!)

aresant16 anni fa

I think this is one of those examples where nobody's rushing in because no matter how you slice it, the product is just not that useful.

I've used it a few times and because the eco system is truly about hiring dirt, dirt cheap labor you kind of get what you pay for.

Beyond that the revenue model sucks - the concept of running millions of micro-transactions and having a little piece of all the action is compelling, but again because the product use is so limited it's difficult to make signifigant revenue.

iwwr16 anni fa

Last I checked, MTurk still accepted only US credit cards. This puts a dent in the potential size of the ecosystem.

If the purpose is dirt-cheap labor, they should open up the door to markets where $cents matter.

thetable16 anni fa

I'm pretty sure I recently used a non-US credit card to pay for MTurk. They require you live in the US, but they don't seem to verify that.

iwwr16 anni fa

They verify for valid zip codes and they prune false addresses from time to time. You have to lie to Amazon, or know someone in the US.

eru16 anni fa

You can work from everywhere. You just have to have a US credit card to pay somebody.

xlorm16 anni fa

It seems like going through CrowdFlower could help a lot. Has anyone used it? I guess it worked for the author.

At my last job I was involved in a project where we were going to use CrowdFlower, but it ended up being canceled. I was always curious about how good their results are.

klous16 anni fa

A worldwide crowdsource platform for the workers who only have access to cell phones is txteagle http://txteagle.com/ and article written on them: http://www.technologyreview.com/printer_friendly_article.asp...

herdrick16 anni fa

Market size. People speculate it's only worth $1M a year to Amazon. But what could it be if it were better done?

ig116 anni fa

I think mturk is vastly underdeveloped, there are all sorts of things people could be using it for, but don't simply because they don't know about it or it's too complex.

Before I started my current startup, creating an mturk competitor was something I considered, but the fundamental problem is handling the payments. Most payment processors wouldn't touch you due to the risk of fraud without a huge (>1mm) deposit.

Paying individuals small amount of money is hard, you pretty much have to use paypal, and if you're lucky enough not to have your account suspended they'll completely rip you on the charges. And you'll still carry all the risk of people trying to use your system for money laundering.