Language HIT

Hello there!

I am a Natural Language Processing researcher planning to post a new set of HITs soon.

I have used MTurk in the past for my MSc thesis, it was a bumpy ride but more or less I am familiar with the platform and want to have a more fair experience for workers as well as ensuring quality.

This time there will be HITs in different languages (spanish, english, french, arabic, chinese, german and italian). My main concern is how is the best way to ensure workers that take the task are fluent in those.

I am aware qualifications can be a headache, and they limit access, and a language test may add too much complexity and time to the process.
Another options is to use country specific Quals, to select that only workers from countries where those languages are spoken are able to take the task. However I am worried there may not be a pool of enough workers in those countries.
Finally I can just make it very specific on the HIT title for which language the task is, and monitor the annotation to filter out those who clearly do not understand the task.

Any suggestions to what would be the best approach is appreciated. The task takes around 1 min per HIT and there will be around 1000 HITs (x3 assignment each) per language. Oh and it doesn't involve any writing, it's 10 yes or no questions.

Thanks!

 

From what I've seen other requesters do when trying to avoid using quals, they just print the material in the desired language and throw in a comprehension check question. One that Google Translate won't get.

 

@Pere-Lluis this is probably your best bet along with making it clear in the hit title. something like French only- hit title of your choice. Putting the language at the front makes it more obvious especially if the workers scraper or whatever is cutting of the end of a long title

 

thank you both! I think I will just make it very clear at the title level and see how it goes. I will also monitor it.

Btw, do you know if it possible/okay to ask for a minimum of HITs to be done in order to get the work accepted? That way I can properly assess in a fair manner if someone really knows what they are doing.

 

Technically no. There are some requesters that do, do that where they will say "must complete X in order to be accepted" and then reject all of the tasks if a worker does less than X... however no good worker touches those hits and when the mass rejections start it gets spread around the worker community pretty quickly killing any chance for the requester to get good work in the future.

From what you've described, probably the best thing to do is put out a batch of say 10 tasks that a worker can do. Pay for all, any, of them that are completed correctly. Give a qualification to those that pass, and release the remaining tasks only to those with the specific quals.

 

Thanks again for the feedback, that was exactly my concern, especially since I do find it a bit unfair to reject work if it's done correctly but just in a lesser amount. However my supervisors thought that would be a good idea.

I like what you suggest but I am a bit afraid it will be hard to get enough workers for some of the languages (such as italian, arabic or chinese). There will be 1000 HITs that take around one minute for each language, and each HIT has to be annotated by 3 different workers. For your suggestion, should I clarify in the title that it is a qualifying HIT? Would it be okay to then specify that all those 10 tasks need to be done?

I convinced my supervisors to raise the payment to 17 cents (initially was 12) per HIT. Do you think that is a fair payment?

 

@ChrisTurk You're better at these type of questions.