Qualifications

Introduction to mTurk Qualifications

Qualifications are a way for Requesters to narrow down the worker pool to those who are most likely to complete their respective HITs accurately and quickly, providing the Requester with optimal results.

With almost 100,000 different qualifications (quals) viewing the mTurk Qualifications tab can be a daunting experience for new Turkers. The majority of these qualifications are inactive or otherwise unnecessary. The various OCMP requesters have over 10,000 different quals, the majority of which are unused.

Default Qualifications

The default qualifications are automatically generated and granted by Amazon Mechanical Turk. They cannot be requested and are a core part of your account. These are different from the Premium Quals granted by the same Author.

QualificationDescription
HIT Abandonment Rate (%)This Qualification is generated automatically and reflects the percentage of HITs which you have accepted but which have expired before you submitted an answer divided by the total number of HITs you have accepted. Your score is a value between 0 and 100. A score of 0 indicates that you have not allowed any HITs to expire before you have submitted an answer for them.
HIT Approval Rate (%)This Qualification is generated automatically and reflects the percentage of HITs for which you have submitted an answer that has been approved divided by the total number of HITs that have been approved or rejected. Your score is a value between 0 and 100. A score of 100 indicates that every HIT you have submitted has been approved.
HIT Rejection Rate (%)This Qualification is generated automatically and reflects the percentage of HITs for which you have submitted an answer that has been rejected divided by the total number of HITs that have been approved or rejected. Your score is a value between 0 and 100. A score of 0 indicates that none of the HITs you have submitted has been rejected.
HIT Return Rate (%)This Qualification is generated automatically and reflects the percentage of HITs which you have accepted and then returned unanswered divided by the total number of HITs you have accepted. Your score is a value between 0 and 100. A score of 0 indicates that you have not returned any HITs.
HIT Submission Rate (%)This Qualification is generated automatically and reflects the percentage of HITs for which you have submitted answers divided by the total number of HITs you have accepted. Your score is a value between 0 and 100. A score of 100 indicates that you have submitted an answer for every HIT you have accepted.
LocationThe Location Qualification represents the location you specified with your mailing address. Some HITs may only be available to residents of particular countries, states, provinces or cities.
Total Approved HITsThis Qualification is generated automatically and reflects the number of HITs which you have submitted an answer that has been approved. Your score is a value greater than or equal to 0.

Requesting Qualifications

Many new users spend time requesting qualifications thinking that a requester will see the request and hopefully grant it. However requesters are never notified of requests. The only way for a requester to see if workers have requested their qualifications is to use the command-line tools or mTurk API, neither of which are user friendly. This makes the practice of requesting qualifications largely fruitless.
Auto-Granted Qualifications
Auto-Granted Quals are the exception to this rule. Some requesters setup their qualifications to automatically assign a score upon request. Some of the more well known examples of this are PicsFromAbove and ProductRNR.
Many will be specially marked as "auto-grant" in the title or description, though this is not universally the case and a worker has no way of knowing for sure until they request the qualification.
PicsFromAbove
PicsFromAbove (PFA for short) is well known to generate a qualification for their batches and set it to auto-grant to workers who request it. They can then complete HITs for the requester until they generate a new qual/batch (at which time the worker must request the newly generated qualification).
ProductRNR
ProductRNR (PRNR for short) is also known to use auto-granted qualifications. However, their system will assign the worker a score for their various qualifications and as the worker completes HITs for the requester their scores will change according to the quality of work they submit.

Its important to note that once a workers score drops below a certain threshold they can no longer complete HITs associated with that qualification.

Qualification Tests

Qualification tests are a quick way for a requester to judge a worker's ability to complete tasks similar to the work they plan on uploading to the platform. They are more limiting than Qualification HITs but are simpler to enable. The important features of a Qual Test a worker should observe before attempting one include:
  • Retake Delay: This is the "cooldown" between attempts at the test. A retake delay of 0 seconds means a worker can immediately try again if they do not pass the test with a sufficient score the first time. If no retake delay is listed that means the worker only has one attempt to pass the test.
  • Timer: The timer is how long a worker has to successfully complete and submit the test before it expires. If a test expires and there is no retakes the worker will be unable to earn the qualification.
  • Auto-Grading: Amazon offers requesters the ability to have their qualification tests automatically graded if there are no text-input fields. There are also ways for Requesters to use the command-line tools and API to auto-grade tests with text-inputs, but they are more involved than the normal method of generating qualification tests.
Non-Disclosure and Other Agreements
A popular use of Qualification Tests is to quickly allow a worker to agree to a non-disclosure or other simple form before granting access to sensitive or private data within a HIT itself.
Amazon Mechanical Turk - Adult Content Qualification
Qualification Test Example using the Adult Content Qualification by Amazon Mechanical Turk

Probably the most important Qualification for a new worker is the Adult Content Qualification. This is technically a test, but it is in fact just a quick agreement to view HITs with adult content on the platform.

Workers should never share normal qualification tests or answers! This qualification was used as an example because of the harmless nature of its questions.

mTurk Qualification Granted message.

When completely properly the worker will be automatically assigned a score of 1 and will be eligible to complete HITs with adult content (such as adult image classification HITs).
https://worker.mturk.com/qualifications/00000000000000000060

Qualification HITs

Qualification HITs serve a similar purpose to Qual Tests, except they allow the worker to demonstrate their proficiency at completing the task itself instead of a simple knowledge test. They also free the Requester from mTurk's limiting qualification test template.

Often these are unpaid, or paid at a much lower rate than the actual work itself. The trade-off is that workers shouldn't have to worry about rejections for completing the work incorrectly or inaccurately since they're both a training tool and proficiency test, though workers should strive to submit their best work possible in order to earn the associated qualification when the Requester processes the data.

Closed Qualifications

Closed qualifications are a term used for qualifications that the Requester is no longer actively granting to new workers. Closed quals are usually coveted because they provide stable work without the competition of unqualled work that anyone can complete. There is less competition for these HITs since the worker pool is naturally limited to those already granted the associated qualification.

Occasionally, as the worker pool naturally dwindles over time, a closed qual may "reopen" for a short period of time (often in the form of Qual HITs) as the Requester looks to recruit new workers. Because of this it is often beneficial to watch requesters who post lucrative closed qual work in case they ever post a qual HIT in the future.

Masters Qualification

Premium Qualifications

Premium Qualifications are a completely voluntary way for workers to provide mTurk (and by extension Requesters) demographic information. Amazon charges a fee for Requesters to apply these Qualifications to their HITs, and as such many Requesters have not adopted them.

These are obtained by filling out the available

Profile Tasks

Profile Tasks, posted by the Amazon Mechanical Turk Team [AXKBO5V1Y75RL] Requester, are the only way to obtain the Premium Qualifications. This is an official Amazon / mTurk account.
Changing Profile Entries

This page has been seen 39,004 times.

Current Discussion: Main discussion

  1. No comments have been posted for this discussion.

Users Browsing This Page (0 members, 1 guests)