CloudResearch is Retiring the “Block Low-Quality Participants” Option

Starting November 1, 2021, CloudResearch will be retiring the Block Low-Quality Participants feature we created in 2020 and continuing with our CloudResearch-Approved Participants option. In this blog, we explain why we’re making this change and what it means for researchers.

Why Are We Removing the Blocked List?
Since Spring 2020, CloudResearch has been engaged in continuous vetting of people on Mechanical Turk. As participants pass through our vetting measures we pair their data with information provided by researchers and make a decision about how likely people are to provide quality data in future studies. People who we think will provide quality data are placed on our Approved List and those who are likely to provide poor data are blocked.

The main reason we are removing the Block Low-Quality Participants feature is because our Approved List is now large enough to handle all studies. Originally, we were concerned that large studies, studies targeting specific groups of participants, or studies with other sampling limitations would be slow to gather data. This is no longer the case. The Approved List now has over 75,000 people and we add new participants each week. As a result, there are enough participants of various demographic groups and levels of experience that the Approved List can support all studies.

Another reason we are removing the Block List option is to better control quality. When researchers run a study using the Block Low-Quality Participants option, everyone CloudResearch has vetted and found to provide suspect data is ineligible for the study (currently over 40,000 accounts!). However, when a study is run using the Blocked List, CloudResearch doesn’t control what proportion of the sample is made up of people on our Approved List or people we’ve never vetted. Because people we’ve never vetted may provide poor data, the Blocked List gives us less control over quality than the Approved List which only includes people we have vetted.

What Effect Will This Change Have on Research?
We do not expect the removal of our Blocked List to affect the viability or speed of research studies. There is also no change in cost to researchers.

As noted above, the Approved List now has over 75,000 participants, and over the last year and a half more than 33,000 studies have been conducted using the Approved List. Supplemental data in our latest preprint show that the demographics of people on the Approved List match those of the U.S. MTurk population well. Thus, we think of our Approved List as an option researchers can use to sample high-quality participants within the US and as a way to restore MTurk to what it was before 2018 when concerns about foreign workers, “bots,” and low-quality data emerged.


Maybe those with CR #2 are going to have their qual removed? Could be good news for some of you, though I'm speculating here given what the above says

 

New Requester Profile!

Lorrie Woods [A185375V40QYT] Survey: Reflecting on your gig work experiences with MTurk (est. 10 min.) - $1.00 | PANDA

Generous

Unrated

Unrated

$29.27 / hour
00:02:03 / completion time

​

Pros:

Cons:

Catchability: Great (HIT Last Seen: 39 seconds ago)
Qualifications:

To read Lorrie Woods's full profile check out TurkerView!​

 

S@TR [A1VHA47ORXU6WR] Would you like to share your opinions about how COVID-19 has affected your happiness? (~26.8 mins) - $2.77 | PANDA

Underpaid

Unrated

Unrated

$5.41 / hour
00:30:42 / completion time

​

Pros:

Cons:
Too long and not worth it.

Catchability: Okay (HIT Last Seen: 22 minutes ago)
Qualifications: 80412233201bcb15666b2ac446860a99b1c3bf42 DoesNotExist ; Total approved HITs >= 500; ab5c70a2a750f5b115e550db4b754b92dfd32213 NotIn 1, 2, 3; HIT approval rate (%) >= 96; Location EqualTo US; 34ee50c9eacb4e076cedf82013242ddbbbff8013 NotIn 0; 3f57dc107e79d59767971dbcaf6078924fa5db0f DoesNotExist ;

To read S@TR's full profile check out TurkerView!​

 

Does this seem like a scam? Hit says it pays $30

 

def scam

 

Yes. I wouldn't risk doing it.

 

I threw it back already. I asked but then remembered I usually go with my gut if the sht doesn't feel right.

 

Lorrie Woods [A185375V40QYT] Survey: Reflecting on your gig work experiences with MTurk (est. 10 min.) - $1.00 | PANDA

Generous

Unrated

Unrated

$25.90 / hour
00:02:19 / completion time

​

Pros:

Cons:

Catchability: Okay (HIT Last Seen: 2 hours ago)
Qualifications:

To read Lorrie Woods's full profile check out TurkerView!​

 

Consumer Lab [AVZ0LVY46FK2J] Consumer Attitudes - $1.00 | PANDA

Generous

Unrated

Unrated

$37.50 / hour
00:01:36 / completion time

​

Pros:

Cons:

Catchability: Great (HIT Last Seen: a second ago)
Qualifications: HIT approval rate (%) > 98; Location EqualTo US;

To read Consumer Lab's full profile check out TurkerView!​

 

Maze [A1D8RNWK41HDVE] Small usability task & feedback on a website/app prototype. (~9 minute) - $1.00 | PANDA

Generous

Unrated

Unrated

$31.30 / hour
00:01:55 / completion time

​

Pros:

Cons:

Catchability: Great (HIT Last Seen: 38 seconds ago)
Qualifications: Total approved HITs >= 500; HIT approval rate (%) >= 95;

To read Maze's full profile check out TurkerView!​

 

Consumer Lab [AVZ0LVY46FK2J] Consumer Attitudes - $1.00 | PANDA

Unrated

Unrated

Unrated

$26.09 / hour
00:02:18 / completion time

​

Pros:

Cons:

Catchability: Great (HIT Last Seen: 6 seconds ago)
Qualifications: HIT approval rate (%) > 98; Location EqualTo US;

To read Consumer Lab's full profile check out TurkerView!​

 

chagall90 [A350544FQSXYOV] Quick opinion survey on politics (~3 min) - $0.25 | PANDA

Underpaid

Unrated

Unrated

$5.81 / hour
00:02:35 / completion time

​

Pros:

Cons:

Catchability: Great (HIT Last Seen: a second ago)
Qualifications: HIT approval rate (%) > 95; Total approved HITs > 50; Location EqualTo US;

To read chagall90's full profile check out TurkerView!​

 

Christina Kan [AI45LKULM0ON3] Consumption & Shopping. Includes an ESSAY WRITING task.(~ 5 minutes) - $0.70 | PANDA

Generous

Unrated

Unrated

$20.49 / hour
00:02:03 / completion time

​

Pros:

Cons:

Catchability: Great (HIT Last Seen: a second ago)
Qualifications: Exc: [66414248-328386] DoesNotExist ; Total approved HITs >= 100; HIT approval rate (%) >= 98; CR Research Group #1 >= 100; Location In US;

To read Christina Kan's full profile check out TurkerView!​

 

New Requester Profile!

Juan [A1R6BNIH55UYZY] Ad Design Test for Tesla Power Bank - $0.10 | PANDA

Underpaid

Unrated

Unrated

$3.67 / hour
00:01:38 / completion time

​

Pros:

Cons:
Two 1-2 sentence writings, one other nonsense question, all for a dime

Catchability: Impossible (HIT Last Seen: 12 minutes ago)
Qualifications: HIT approval rate (%) > 95; Location EqualTo US;

To read Juan's full profile check out TurkerView!​

 

Christina Kan [AI45LKULM0ON3] Consumption & Shopping. Includes an ESSAY WRITING task.(~ 5 minutes) - $0.70 | PANDA

Unrated

Unrated

Unrated

$19.24 / hour
00:02:11 / completion time

​

Pros:

Cons:

Catchability: Great (HIT Last Seen: a second ago)
Qualifications: Exc: [66414248-328386] DoesNotExist ; Total approved HITs >= 100; HIT approval rate (%) >= 98; CR Research Group #1 >= 100; Location In US;

To read Christina Kan's full profile check out TurkerView!​

 

Good morning

 

Maze [A1D8RNWK41HDVE] Small usability task & feedback on a website/app prototype. (~9 minute) - $1.00 | PANDA

Unrated

Unrated

Unrated

$10.43 / hour
00:05:45 / completion time

​

Pros:

Cons:

Catchability: Okay (HIT Last Seen: 21 minutes ago)
Qualifications: Total approved HITs >= 500; HIT approval rate (%) >= 95;

To read Maze's full profile check out TurkerView!​