I am having trouble trying to grasp how to create a qualification test with the mturk requester developer feature. I have a very simple line of code to be created. Can someone assist me with coding my qualification test.?

I need to be able to ask a set of questions: "What is the url of the given link?", "Is it a software or a game located in the link given", "How many downloads, on the website where the link is located, does it currently have?"

I need these three questions to be answered.

I have multiple tasks that need to be done but if I can get some type of code that is one iteration of the questions I need asked and answered, I can just copy it over and over to the amount that I desire. Can someone help me create this code?

 

@ChrisTurk

 

IMO, the dev tools for qualification tests are bootycheeks. It would be worlds easier to put out HITs to use as a qualification ("qual HITs"), then grant the qualification to those who perform satisfactorily.

But if you're set on the qualification test feature, someone more intelligent than I (@ChrisTurk or @Tjololo perhaps) might be able to help you.

 

The terminology you're using here is really confusing.

Are you trying to qualify a certain # of workers to complete the 3 question task, or are you trying to create a custom qualification task with three questions?

If the former, you'll just create it as if its a normal task but release a small # of HITs - preferably all with answers you already know so that you can "grade" workers accordingly.

So lets say you've got 15,000 HITs that need to be completed, a qual test for those might be 500 or so of the 15,000 that you do yourself (so you have the answers) then release them (without the answers) - have workers complete 5-10 each and then create a qualification and grant it to the workers who answer the HITs correctly.

The other version of a qualification test can be implemented through the API, but you cannot dynamically create the content like a qual HIT.. are you trying to generate the code for that? It can only be interfaced through the CLI or API, not the web dashboard, so kinda hard to grasp what you're going for here.

 

I am trying to achieve this via API because I need to gauge how long on average it takes an individual to do the job. So I need to set specific questions and answers. If I can do that via web dashboard I don't need the API. It is just that I don't know what either or can provide for me in regards to getting my three questions answered.

 

The API has nothing inherent to it that will solve this problem. MTurk will give you a general idea of how long a worker had the HIT in their queue but it will not give you an accurate estimate of how long the worker actually spent on the task. You will either need manual feedback from workers or some javascript snippets to judge how long workers are active in the frame.

Your problem does not have a good pre-built solution to it in MTurk, more or less.

The web UI is simpler than the API. You can put the inputs for all 3 questions into a CSV file, upload it, and it will populate one of MTurk's pre-built templates with your values. Essentially, its simpler than programming your own interface w/ the API. But the API offers more control in things you can do if you have the time to spend with it.

You could also host your 3 questions on your own server with whatever custom code you'd like and just link it as an external HIT, that requires the API interfacing though IIRC.

 

How would I go about using the Web UI and CSV file combo to get my questions up? Where is a tutorial for that?

 

MTurk has published documentation available:

Web UI: https://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/Welcome.html
API: https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/Welcome.html

• This page explains how you can use embedded variables to offer dynamic content in their pre-built templates (ie, image links, urls, etc):
  • https://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/CreatingaHITTemplate.html
• This page specifically details how the CSV file upload works:
  • https://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/PublishingYourBatchofHITs.html

But basically running through the docs top to bottom will give you a general overview of the steps to publishing a HIT.

 

So, in regards to CSV file, how would I structure that file for it to be read by Mturk. I know what details on want within the file but I am under the impression that there is a special way to set up the file as with most computer documents, the way each of them is read is unique. I see no info on how to develop the CSV for usage on mturk. How do I create my CSV file?

 

This link goes over that exactly, step by step even: https://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/CreatingaHITTemplate.html

name the variable: ${image_url}

In the CSV file, title a column image_url then all the data in the rows below the column header will be embedded in the generated HIT template. One line of data for one HIT. A single HIT can be completed by however many workers you specifiy in the project details for "Number of assignments per HIT"

 

I used the web UI and got my .CSV uploaded and created my hit with my 3 categories of questions. How do I know if the hit was done correctly creation wise? I can't seem to access a preview of the hit or a live version of it. I created the hit in the requestersandbox

 

https://workersandbox.mturk.com

You can log in with the same credentials as your requester account & search for your HIT & complete it and everything that way you can be sure it works properly and the data you get back in the requester dashboard is correct.

 

For my test hit I gave it a Master's Qualification. Now I can't access the hit on workersandbox.com. How do I get rid of the hit qualification so I can access the hit on workersandbox.com? When I create a new test hit it doesn't allow me to input things like price of the task initially.

 

Gotta go to the "Create" tab and edit the project which will bring you back to the area where you can add/remove quals, etc. You'll have to cancel the old batch & republish after making your changes.

FWIW Masters on production is going to be very hit or miss. You're paying a premium for something that likely isn't worth it. You're better off using Location is US, >5,000 approved HITs, and >98% approval rate. You can research around and find tons of in-depth explanations on why but that's the short advice I can throw your way.

 

Thanks again man you are a lot of help to me. Just wanted to make that clear

 

It is worth mentioning that if you have some sort of weird desire to use the masters qual you can get around the fee by paying via bonus or giving workers a qual test with the masters qual and not have masters on the actual work.