Hi everyone! We're a nonprofit that promotes open access to scholarly research, and we use Mechanical Turk to check the quality of the results in Unpaywall.

Results have been good in general but there are some common cases where most workers give the wrong answers, and it would take more instruction to explain than anyone is willing to read for a 10 or 15 cent HIT (understandably). We're looking for a level of accuracy that would take a couple of hours of upfront training, and I'm looking for feedback on this plan to set it up so it pays fairly and people can tell what's going on:

• Create a qualification that requires a detailed test, accompanied by about 10-20 pages of examples and explanation. There are enough tricky cases that it will probably take more than one attempt at the test.
• Create a batch of do-nothing HITs (click button, get paid) that require a certain score on the qualification. Not perfect, but enough that you have to try. These would pay maybe $15-$20 and could only be completed once.
• Require a higher score, near perfect, on the qualification for the real batches we run later on. Mention the do-nothing screener batches in the description so it doesn't look like we're paying 5 cents an hour.

The idea is if you spend time reading up and try the test a couple of times and it's not clicking, hey, at least you got paid to take it.

Does that sound like it would work? Too complicated? Any other suggestions?

Thanks!

 

Depending on how much work you have to post you may get a fair reception. I think giving people the opportunity to qual and opt out if they can't get it to click is optimal. We all have different preferences for the type of work we like to complete. I think a lot of it will come down to having the right people know about and completing the qual. If the task is so difficult that a majority of workers are making the same mistake or getting the same items wrong, perhaps looking in to the cause of that might be good. The thing many workers fear is a rejection, especially on batch work and with new to the platform/new to them requesters. I think you have some solid ideas about how you would like to progress. Hope to catch your qual task when it hits, maybe running it a few times and at different times of the day will attract a wider pool of talent for you :)

 

Nice, thanks! I'm going to scatter the qual runs over a couple of weeks for that reason.

I spent a lot of time tweaking the instructions and examples in previous batches, and I think it's not a matter of being difficult, but of needing more context than we can expect people to absorb for what might only be one task. We're obviously not the biggest requester, but between the volume we're planning on and the pay for the qual it should be worth trying for a lot of people.

 

You are not going to have any problems finding workers willing to learn what is required to do the work you're requesting under the assumption that similar work from the past is posted.

Construct whatever is needed to ensure that a worker at the end of it is knowledgeable enough to handle your edge cases, and then throw a qualification and work at them.

Explain fully, as you did here, what is going to be required in order to learn everything and what you have planned for those who choose to go through the training. Because of what that account has posted in the past, I surely don't see a problem in finding a group of workers willing to spend the time to learn this work.

 

agree with everything said here. Might help to post here when you are posting the qual batch so it is not missed especially if it initially looks like it is paying much less than it is.

 

I agree with everything that's been said here. I just wanted to add if you are posting your hits over time that it's really nice if you make a Google Doc or something searchable that is linked in the hit. So if I haven't done your hits in a while I can review the instructions and it's also nice to be able to do a little Ctrl-F'ing in the document for when I hit an edge case and want to make sure I'm doing things right. Alegion does this and it works out pretty well for me.

 

Hi again, thanks for the comments! We just posted a batch if anyone is interested. This is a test batch of 25 assignments so they might be gone already. https://worker.mturk.com/requesters/A34MJ5ATWO4JEI/projects?ref=w_pl_prvw

It's pretty much as described in the OP: take the test and if you get 80% (which should be really easy if you read the instructions) you get to do the HIT and get paid. I think the potentially confusing thing is that we want a near-perfect qual score for the "real" tasks we post later, but I don't want to stiff anyone who's making an honest attempt so there's a lower threshold here. There's limited space to explain this until you get into the test, but I tried my best to cram it into the description.

 

Thanks for the high quality HITs.

 

I liked doing your HITs in the past. Can you please notify us when you post another batch of qualification HITs? Thanks!

 

@ChrisTurk when r u gonna verify this requester account

 

Probs in the morning, they put in the request Friday but I usually do them in rounds to make it easier

 

:emoji_thinking::emoji_thinking::emoji_thinking::emoji_thinking: