We hire a pool of Turkers for a large scale AI annotation task

Discussion in 'Requesters' started by CMU-LTI Crowsourcing009, Apr 24, 2019.

  1. CMU-LTI Crowsourcing009

    CMU-LTI Crowsourcing009 New Turker

    Messages:
    5
    Ratings:
    +11
    Hello all,

    We are Carnegie Mellon University researchers working on the intersection of Natural Language Processing, Computer Vision, and Machine Learning.

    We have this new cool task where one person hides Waldo in panoramic images and provides a list of clues to find him, and another one is trying to find Waldo with those clues. Before we run a large scale study we would like to hire a pool of annotators.

    So far we ran 2 pilot studies and thanks to Turkers we are able to improve it substantially. We fixed lots of bugs and compensate Turkers time for those bugs. Based on the average task completion time we keep readjusting the reward. For the large scale study, we plan to have a leaderboard for "the best hiders" and "the best finders". Top hiders and finders will receive extra bonuses.

    HIT Information:
    Requester: Carnegie Mellon 009
    Title: Hide Waldo!
    Reward: 2.25$
    Expected HIT length:  10-15 minutes
    Qualifications Required:  HIT Approval Rate > 90, Location is the US/UK

    We are looking forward to your feedback! For further information please reply to this thread, DM us in this forum, or email us lti-crowdsourcing-009@andrew.cmu.edu
     
    • Like Like x 2
  2. angel

    angel Survey Slinger

    Messages:
    15,286
    Gender:
    Female
    Ratings:
    +28,535
    I found the hit but it also requires Masters so I can't do it
    upload_2019-4-24_9-6-30.png

    FYI
    setting the qualifications something like 5,000 approved and HIT approval rate (%) is not less than 99 often gets better results than the master qual just remeber Amazon rounds down so make sure you include 99%. In other words if it is set for only >99% that is the same as requiring100%
     
    • Like Like x 1
    Last edited: Apr 24, 2019
  3. InfiniteChanges

    InfiniteChanges Survey Slinger TurkerView Masters

    Messages:
    8,880
    Gender:
    Male
    Ratings:
    +9,937
    Agreed. Masters is something that is limiting your potential pool of workers.
     
    Last edited: Apr 24, 2019
  4. ChrisTurk

    ChrisTurk Administrator

    Messages:
    56,679
    Ratings:
    +162,927
    Just curious are you guys associated w/ the HCI lab at CMU?
     
  5. ChrisTurk

    ChrisTurk Administrator

    Messages:
    56,679
    Ratings:
    +162,927
    A few folks got it just a week ago lol. Still limits the worker pool way too much to use for a large scale study IMO so your second half is probably correct but they do still replenish the pool at least a bit nowadays.
     
    • LOL LOL x 1
  6. CMU-LTI Crowsourcing009

    CMU-LTI Crowsourcing009 New Turker

    Messages:
    5
    Ratings:
    +11
    Thanks all! We are not pros like HCI people, new in the annotation stage, that's why we are making these rookie mistakes :)

    The next batch will not have masters and we will readjust qualifications!
     
    • Like Like x 6
    • Today I Learned Today I Learned x 1
  7. slothbear

    slothbear Survey Slinger

    Messages:
    10,821
    Gender:
    Male
    Ratings:
    +22,069
    Oh, don't worry about removing masters, I've got it, and I'm all you need ;) j/k

    HIT is laid out well and pretty straightforward. Can't think of many improvements at the moment.
     
    • LOL LOL x 2
  8. angel

    angel Survey Slinger

    Messages:
    15,286
    Gender:
    Female
    Ratings:
    +28,535
    :hitwithrock:
     
    • LOL LOL x 4
  9. StuffAlleySays

    StuffAlleySays Survey Slinger

    Messages:
    3,313
    Gender:
    Female
    Ratings:
    +4,096
    I'll be there once Jedi status is lifted.
     
    • LOL LOL x 2
    • Like Like x 1
  10. CMU-LTI Crowsourcing009

    CMU-LTI Crowsourcing009 New Turker

    Messages:
    5
    Ratings:
    +11
    Folks, we just published another batch of HITs, now we do not have masters requirement. We are looking forward to working with you and hear your feedback!
     
    • Like Like x 1
  11. InfiniteChanges

    InfiniteChanges Survey Slinger TurkerView Masters

    Messages:
    8,880
    Gender:
    Male
    Ratings:
    +9,937
    It looks like there's some wrong with the qualification.

    http://prntscr.com/nh8hyw

    [​IMG]
     
  12. angel

    angel Survey Slinger

    Messages:
    15,286
    Gender:
    Female
    Ratings:
    +28,535
    what did you search for? I searched the requestor name and the hit title and it's not showing up for me
     
  13. InfiniteChanges

    InfiniteChanges Survey Slinger TurkerView Masters

    Messages:
    8,880
    Gender:
    Male
    Ratings:
    +9,937
    I just searched "mellon". Make sure you have "I'm qualified to work on" unchecked in your filter.
     
  14. angel

    angel Survey Slinger

    Messages:
    15,286
    Gender:
    Female
    Ratings:
    +28,535
    hmmm :dunno:
    upload_2019-4-26_10-15-40.png
     
  15. CMU-LTI Crowsourcing009

    CMU-LTI Crowsourcing009 New Turker

    Messages:
    5
    Ratings:
    +11
    Could you check again?
     
  16. InfiniteChanges

    InfiniteChanges Survey Slinger TurkerView Masters

    Messages:
    8,880
    Gender:
    Male
    Ratings:
    +9,937
    Was able to catch it now. Thank you.
     
  17. angel

    angel Survey Slinger

    Messages:
    15,286
    Gender:
    Female
    Ratings:
    +28,535
    still not showing up for me
     
  18. BluAngel

    BluAngel Well-Known Turker

    Messages:
    2,051
    Gender:
    Female
    Ratings:
    +1,046
    I cannot find it and "I'm qualified to work on" unchecked in my filter.
     
  19. angel

    angel Survey Slinger

    Messages:
    15,286
    Gender:
    Female
    Ratings:
    +28,535
    @InfiniteChanges must have the magic touch this morning
     
    • LOL LOL x 1
    Last edited: Apr 26, 2019
  20. InfiniteChanges

    InfiniteChanges Survey Slinger TurkerView Masters

    Messages:
    8,880
    Gender:
    Male
    Ratings:
    +9,937
    I was able to find, catch, and complete the task... but man was that a pain on such a smaller work area.
     
    • Like Like x 1