New Requester: Seeking Advice on How to Limit a Batch to One Worker

Discussion in 'Requesters' started by liam_w, Nov 13, 2018.

  1. liam_w

    liam_w New Turker

    Messages:
    7
    Gender:
    Male
    Ratings:
    +10
    Hi everyone,

    Brand new requester here. We have a large number of stores which we’d like to score on attributes that impact their performance (visibility, ingress-egress, etc.). There is some subjectivity involved in this scoring, so we would like to set up a feedback loop to help the worker align with how we would score a store.

    Because of this learning curve we’d like to keep the number of workers limited and allow them to work on a large number of stores. We would start small, with five HITs in a batch, and give the worker feedback on their scoring. How can we set up our request so that one worker completes all of the HITs in a batch rather than the HITs being split out among multiple workers?

    Thanks for your help!
     
    • Like Like x 3
  2. CorvusSapiens

    CorvusSapiens Survey Slinger

    Messages:
    6,777
    Gender:
    Male
    Ratings:
    +9,895
    You can assign a qualification to the worker and set the hits to require that qual.

    This page should have info on doing that: https://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/AssigningaQualificationTypetoaWorker.html
     
    • Like Like x 3
  3. TurkedOut

    TurkedOut Survey Slinger

    Messages:
    8,241
    Gender:
    Male
    Ratings:
    +14,450
    Corvus hit it right on the head. Use custom quals to limit workers.
     
  4. ChrisTurk

    ChrisTurk Administrator

    Messages:
    56,679
    Ratings:
    +162,927
    The initial setup is going to be the difficult part b/c you can't natively ensure a worker can actually get 5 HITs. You also wont really be able to train them, you're better off using a system where you just recruit the workers who happen to intuitively understand your instructions/needs and/or "guess" correctly on qualification HITs.

    OCMP5 has a setup similar to what you want. What they do is just release "qualification" HITs periodically. A worker has to complete 10 HITs (ALL HITs have a pre-known answer so the worker can be graded) and they're scored. If they get 9/10 correct they get the qual and can do the work. Yes, some people get lucky (or unlucky) and qualify then do crappy work & vice-versa some folks get screened out who would have been really good but its the quickest/dirtiest initial screening system.

    The big downside here is that you can't "reserve" 10 HITs for each worker, so you essentially have to release exponentially more HITs than you intend to actually qual people for. Where you'd expect to only release 10 HITs per worker you want to get the qual you'll actually have to release far more because some workers will take 3, others 1-2, and then others will be "stuck" having only 5-6 HITs and never get their score.

    I can't really give you advice on exact #s you have to release because it will all depend on the popularity of your requester account and if workers share the qual with social networks like this, etc etc. Make a plan to keep their score across batches (you'll need to do external HITs for this) that way if you don't get enough workers in the first pass you can just release more HITs until you do.

    After that you can start with feedback looping your worker base. But honestly that particular problem is one of MTurks weak points, its better suited to projects that can use consensus to determine accuracy there are no great ways of "training" your workforce on MTurk so its going to have to be something you guys build out for your requirements.
     
    • Like Like x 1
  5. liam_w

    liam_w New Turker

    Messages:
    7
    Gender:
    Male
    Ratings:
    +10
    Thank you for your advice. Based on your insights and some additional research, I’m contemplating the following approach to enable a worker to address multiple HITs at a time. Can you tell me whether any of this is out of line?

    First, I thought I might ask whether anyone on this forum is interested in participating. The pilot HIT is in the MT sandbox under the Requester Dena Whitaker. Anyone who checks it out and is interested may respond here with their worker ID.

    Second, I would group the five stores for the pilot into a batch of HITs. I understand that if I keep the metadata the same for all of the HITS then they will be grouped together for the worker. I will create a custom qualification for these HITs (say, “Store Classifier 1”) and assign one of the workers who has responded in the forum to this classification. It seems like this would limit all five HITs in the batch to that one worker.

    Third, I would create a second, third, fourth and fifth batch of five HITs apiece. Each of these batches would be slightly differentiated by the description (Store Classification 2, 3, 4, 5) for the HITs within them. Then, like the first batch, I would create a custom qualification for the HITS within each (say, “Store Classifier 2, 3, 4, 5”) and assign one of the workers who has responded in the forum to each of these classifications.

    I will not reject any work which appears to have been done in good faith. In the sandbox the reward is listed as $2.50, but that’s the “long term” rate. For this first set of batches I’ll set the rate at $3.50 to help offset the time to go through the instructions. For workers who have done a good job and whom I’d like to offer further batches to do, I will give a $.01 bonus. In the message accompanying the bonus I will give feedback on how the worker’s scoring for each store aligned with our scoring.

    I hope to develop a core of workers who can devote a significant amount of time to this project. At the end of the project I will also give some bonus to those who make significant contributions.

    Thanks!
     
  6. slothbear

    slothbear Survey Slinger

    Messages:
    10,821
    Gender:
    Male
    Ratings:
    +22,069
    I may be incorrect, but I think you can set a limit on the number of HITs a worker can accept in a batch. Say you assign 5 workers with the "storeClassifier' qual, then post a batch of 25 stores with the max accept limit to 5 per worker... @ChrisTurk can correct me if I'm wrong.
     
  7. A6_Foul_Out

    A6_Foul_Out Organic Meme Panda TurkerView Masters

    Messages:
    17,777
    Gender:
    Male
    Ratings:
    +23,954
    They have to have a qual first, I think CT was referring to posting a qual batch prior to that?

    And there isn't a solution to ensure people take all 5 hits if he just posts the batch open to all that isn't very wasteful.

    ex. Andreas burning money on every qual test where people only do 1 of the 5 etc.
     
  8. A6_Foul_Out

    A6_Foul_Out Organic Meme Panda TurkerView Masters

    Messages:
    17,777
    Gender:
    Male
    Ratings:
    +23,954
    @slothbear
    do you happen to know the requester ID?
     
  9. slothbear

    slothbear Survey Slinger

    Messages:
    10,821
    Gender:
    Male
    Ratings:
    +22,069
    Oh yeah, that's kind of what I meant. Give 5 workers the qualification somehow...known worker ids, qual test, etc.., then drop a batch for them.


    Reading back, do you mean there isn't a way to ensure the qualled workers do all the work? Maybe move some of the per HIT pay to a bonus after completing the 5 HITs...a half now, half later type of thing...sorry, just woke up from a nap and my brain is still fuzzy...


    A3VGBHJ6N2NQYF

    sandbox link:
    https://workersandbox.mturk.com/requesters/A3VGBHJ6N2NQYF/projects?ref=w_pl_prvw





    Anyway @liam_w, I wouldn't mind giving them a try when it's time.
     
    • Today I Learned Today I Learned x 1
  10. A6_Foul_Out

    A6_Foul_Out Organic Meme Panda TurkerView Masters

    Messages:
    17,777
    Gender:
    Male
    Ratings:
    +23,954
    nah, not qualled workers, unqualled workers.

    If he posts 500 hits without quals, 5 hits each for 100 workers that's just going to be a free for all and a smaller percentage of workers are actually going to get 5.
     
    • Like Like x 1
  11. Ezridax

    Ezridax Well-Known Turker

    Messages:
    2,014
    Gender:
    Female
    Ratings:
    +1,795
    Ouch. I can’t even do the sandbox hit. The preview looks interesting though. (I have apparently only done 3 sandbox hits)
     
  12. liam_w

    liam_w New Turker

    Messages:
    7
    Gender:
    Male
    Ratings:
    +10
    Hi Ezridax. I don't quite understand your point on not being able to do the sandbox hit. Are you saying that if a worker hasn't met the minimum required hits in the sandbox (as opposed to having done the minimum number of hits in the published hits section) that they can't try out the hit in the sandbox?

    Thanks,
     
  13. Ezridax

    Ezridax Well-Known Turker

    Messages:
    2,014
    Gender:
    Female
    Ratings:
    +1,795
    Yes. That's what I'm saying. I've completed well over the required number on the main platform, but in the sandbox, I've submitted 3 hits.
     
  14. Zorseshoes

    Zorseshoes Well-Known Turker

    Messages:
    2,006
    Gender:
    Female
    Ratings:
    +1,252
    I am having the same experience as @Ezridax . The sandbox HIT says I am not qualified because I do not have 1000 HITs--I have way more than that number on Mechanical Turk, but only 1 HIT on the sandbox subsite.
     
    • Love Love x 1
  15. liam_w

    liam_w New Turker

    Messages:
    7
    Gender:
    Male
    Ratings:
    +10
    Hi. We have redone the HIT in the sandbox with the required number of HITs set at "> 0". Is it possible to do the sandbox HIT now?
     
  16. liam_w

    liam_w New Turker

    Messages:
    7
    Gender:
    Male
    Ratings:
    +10