10/02 - TurkerView - Return That HIT! Warning/Review API

Discussion in 'Daily mTurk HITs Threads' started by ChrisTurk, Oct 2, 2018.

Thread Status:
Not open for further replies.
  1. ChrisTurk

    ChrisTurk Administrator

    Messages:
    56,679
    Ratings:
    +162,927
    Hey folks,

    So this is a new approach for TV entirely. I feel like nowadays the website does a good job at collating/displaying data about Requesters to visitors who need it. The API generally provides folks a quick glance at the pay ratings/history of a Requester.

    What we don't have is a good heads-up system of identifying all the crap we invariably throw back into the pool from day to day. TV has always been restricted to completed HIT reviews only - which has been very good for data quality but has definitely left some blind spots in the dataset.

    Starting today you'll no longer have to fully complete a HIT in order to leave a note for other workers on why it isn't worth finishing.

    Returned HIT Reviews
    These are gonna be kind of weird since they wont go directly to the website, instead they'll only be displayed on the task completion page for a HIT in your queue.

    The upside is that instead of having to finish a really crappy (or impossible) HIT in order to leave a note for others folks TurkerView will now facilitate an on-site communication network between workers to provide a heads up display about HITs that deserve a warning note to incoming users.

    In order to try to keep data quality standards close to those for the on-site reviews these are tied to the action of returning a HIT out of your queue. This ensures that users who are leaving the reviews have at least looked over the HIT (probably even started doing it) before deciding it is indeed so bad its not even worth submitting.

    The UI on the task/HIT page is fairly simple:
    [​IMG]

    The Return Report Form
    [​IMG]


    I started off by keeping this as simple as possible. A few checkboxes, an optional comment box & you can get on with your day. No need to leave MTurk or even your work queue, as soon as you submit the data TVJS will finish returning the HIT for you and MTurk will forward you along to the next HIT in your queue to keep you working & wasting less time. Anyone who loads the HIT after you return it & has TVJS installed with get a notification that the HIT has been reported as bad & they can give it a looksie.

    In keeping with the "waste less time" theme we also don't want to force workers to come to the website to get a heads up on bad HITs:

    The Returned Report Overview
    [​IMG]

    The red warning buttons as seen in the top image will only appear on HITs that have active reports, when they do exist they'll pull up the above overview page of reports w/ comments (where available) and some stats about why workers returned the HITs previously.

    The overview will show you how many users reported each specific category. On top of that if there is an unpaid screener in the HIT their progress in minutes/seconds will be displayed so you can make an informed decision about how long the screening process for a HIT can take. If most folks get screened out in 20-30s for a $5 HIT, you may be willing to continue on.

    Alternatively, requesters like Pirc should quickly gain a reputation for their annoying screening practices. Either way, the data is simply there to help you make an informed decision on the HIT :)

    We also provide comments (where available) by users. Some of these comments are auto-generated by TV (in cases where users mark a HIT as underpaid, it'll show you their hourly at the time they decided to bail out of the HIT). While not perfect, that along w/ comments will hopefully provide a little insight into the whys they returned the HIT.

    Again, none of this data will be fed to the website directly & it will not affect API based ratings at the moment. Its an entirely new system designed for a purpose TV previously has not been equipped to handle. I personally think this system will be a little more resilient to "noise" since much of it is fluid data but some larger conclusions/additions to requester profiles will likely be in the works based on it.

    TurkerViewJS 7
    Aside from the above changes TVJS has gotten some QoL love:
    • The wage tracker now has "sticky nav" attributes and will pop around to the upper right corner of the screen when you're working on a HIT.
    • The wage tracker will now update the titlebar tab title to indicate the wage/completion timer so you can still see it when completing another survey.
      • Because so many other scripts also have this functionality you can toggle it off by clicking on TurkerView in the upper nav area to bring up the dashboard and going to the settings (click the gear icon in the upper right of the modal)
    • Minor visual changes to certain popovers/tooltips/etc

    Feedback, Questions, Suggestions, Etc
    This was a quick sprint build over the weekend based off a short reddit convo. I honestly did not have any of this planned or mocked up so there will be gaps in coverage, missing checkbox categories workers want, etc etc.

    I'm sure there will also be bugs or missing/incomplete features once it goes through the grind. Please just @me on the forum or DM me.

    But in general any bugs/snafus you hit, improvement avenues you see, etc etc as always immensely appreciated.
     
    • Love Love x 14
    • Like Like x 8
    • Today I Learned Today I Learned x 2
    • Nom Nom Nom! Nom Nom Nom! x 1
  2. urtho9

    urtho9 Active Turker

    Messages:
    950
    Gender:
    Male
    Ratings:
    +709
    [​IMG]
     
    • LOL LOL x 1
  3. turker

    turker Survey Slinger

    Messages:
    13,565
    Ratings:
    +21,075
    [​IMG]
    good morning let's make this the best tuesday ever!
     
    • Nom Nom Nom! Nom Nom Nom! x 3
    • Like Like x 1
  4. Bloop

    Bloop Well-Known Turker

    Messages:
    2,365
    Gender:
    Female
    Ratings:
    +2,652
    Title: THOSE BELONGING TO MINORITY GROUPS AND FEMALES ENCOURAGED TO RESPOND: Answer a survey about the atmosphere of cafes | PANDA
    Requester: Amelie Burgess [A1AK189WS4GKNZ]
    TurkerView: [ $25.85 / hour ]
    Description: The project’s aim is to gain an understanding of how culture influences the perceptions of atmospherics in the servicescape. This understanding will be used to examine if and how these perceptions affect behaviors and responses.
    Duration: 01:00:00
    Available: 1
    Reward: $2.50
    Qualifications:
    • None
    [3RB76ORN6TG6830VBUSGH9JM3QQEE6]

    I can't review this one but it took me about 5 minutes.
     
    • Nom Nom Nom! Nom Nom Nom! x 3
  5. NobodyImportant

    NobodyImportant Well-Known Turker

    Messages:
    2,744
    Gender:
    Male
    Ratings:
    +2,698
    At first glance I was not sure that I was a big fan of this feature, but after considering the entire post it seems like it could be quite useful.
    I'm a bit stupid, so it will take me awhile to wrap my head around the change, but either way Thanks for the hard work
     
  6. NobodyImportant

    NobodyImportant Well-Known Turker

    Messages:
    2,744
    Gender:
    Male
    Ratings:
    +2,698
    Minorities and women get all the good surveys, what BS
     
    • LOL LOL x 2
  7. EmTurker

    EmTurker Active Turker

    Messages:
    455
    Gender:
    Male
    Ratings:
    +353
    I know I'm sick today and time sure feels like it's staying still but I'm confident it's the 2nd and not the 1st of October :D
     
    • LOL LOL x 1
  8. ClassicChump

    ClassicChump Well-Known Turker

    Messages:
    2,363
    Gender:
    Male
    Ratings:
    +3,051
    [​IMG]
     
    • LOL LOL x 2
    • Like Like x 1
  9. Kayashi

    Kayashi Titler

    Messages:
    7,279
    Gender:
    Male
    Ratings:
    +14,908
     
    • Like Like x 2
  10. NobodyImportant

    NobodyImportant Well-Known Turker

    Messages:
    2,744
    Gender:
    Male
    Ratings:
    +2,698
    C'mon man, I was hoping to get a bite :O
     
  11. KaraStarbuck

    KaraStarbuck Active Turker

    Messages:
    133
    Gender:
    Female
    Ratings:
    +81
    So. The student friendship $3 hit. It says the average bonus is between $5-$8. Cool. Says it takes 45 minutes. I do the hit. Takes over an hour. Fine. I earn a grand total of $1.50 bonus. Fine, I'm an idiot. Fill out the survey and submit. It says debugging task complete, if this was a real hit it would now submit to amazon turk. So I just wasted an hour on a hit I can't submit. The only reason I am not having a screaming fit at the moment is my roommate is sleeping and he gets cranky when I yell at the internet in the middle of the night,
     
    • Today I Learned Today I Learned x 2
    • LOL LOL x 1
  12. KaraStarbuck

    KaraStarbuck Active Turker

    Messages:
    133
    Gender:
    Female
    Ratings:
    +81
    I just realized I can submit the hit. And that it says the minimum bonus is $3. Life is wonderful.
     
    • Like Like x 2
  13. NobodyImportant

    NobodyImportant Well-Known Turker

    Messages:
    2,744
    Gender:
    Male
    Ratings:
    +2,698
    Robot Ethics and Aerial Vehicles Lab [A3C79NCG9MFEH8] Robot Experiment in Emergency Scenarios - $3.00 | PANDA


    Unrated

    Unrated

    Unrated
    $15.65 / hour
    00:11:30 / completion time
    Pros:
    A easy game
    Cons:
    Cannot copy mturk id/completion code
    "game" took forever to load
    An obnoxious amount of threats of rejection

    Qualifications: Total approved HITs >= 100; HIT approval rate (%) >= 90; Location EqualTo US; Single Attempt DoesNotExist ;
    To read Robot Ethics and Aerial Vehicles Lab's full profile check out TurkerView!
     
    • Nom Nom Nom! Nom Nom Nom! x 10
    • Good Review! Good Review! x 1
  14. carrie35

    carrie35 Turker

    Messages:
    33
    Gender:
    Female
    Ratings:
    +23
    Did we loose a day/24 hours? My calendar says 10/2. Or is this not the daily thread? Thank you. If we could turn back time ... no.
     
    • LOL LOL x 1
  15. NobodyImportant

    NobodyImportant Well-Known Turker

    Messages:
    2,744
    Gender:
    Male
    Ratings:
    +2,698
    Gee thanks , now I have that terrible song in my head
     
    • Like Like x 1
  16. Vince

    Vince Turker

    Messages:
    229
    Gender:
    Male
    Ratings:
    +352
    Alex [A193ITC4S0HPC4] Measuring your internet experience - $1.50 | PANDA


    Unrated

    Unrated

    Unrated
    $42.86 / hour
    00:02:06 / completion time
    Pros:
    Cons:
    Have to install chrome add on

    Qualifications: Masters Exists ;
    To read Alex's full profile check out TurkerView!
     
    • Nom Nom Nom! Nom Nom Nom! x 4
  17. Vince

    Vince Turker

    Messages:
    229
    Gender:
    Male
    Ratings:
    +352
    Devin Fidler [A2HIKOY3S9L9O3] Brainstorm Names for a Project Focused on Extreme Innovation - $0.25 | PANDA


    Generous

    Unrated

    Unrated
    $13.04 / hour
    00:01:09 / completion time
    Pros:
    Cons:

    Qualifications: Masters Exists ;
    To read Devin Fidler's full profile check out TurkerView!
     
    • Nom Nom Nom! Nom Nom Nom! x 1
  18. SAJ

    SAJ Survey Slinger

    Messages:
    27,495
    Gender:
    Male
    Ratings:
    +51,479
    Jollibee spaghetti at 1:37 a.m. is the adultest life
     
    • Nom Nom Nom! Nom Nom Nom! x 1
  19. Mothra

    Mothra Survey Slinger

    Messages:
    6,468
    Ratings:
    +11,952
    Junsub Yi [A1R8M99IAULKFJ] Perceptions and Emotions about IoT Device(Smartwatch) - $1.20 | PANDA


    Unrated

    Unrated

    Unrated
    $11.11 / hour
    00:06:29 / completion time
    Pros:
    Fair compensation
    Cons:
    Trapped forever in the bubble mines

    Qualifications: Old participants_smartwatch DoesNotExist ; HIT approval rate (%) > 95; Location EqualTo US;
    To read Junsub Yi's full profile check out TurkerView!
     
    • Nom Nom Nom! Nom Nom Nom! x 1
  20. SAJ

    SAJ Survey Slinger

    Messages:
    27,495
    Gender:
    Male
    Ratings:
    +51,479
    Roomy got back from Montreal tonight...I got a couple hoodies and fridge magnets :yesyes:
    [​IMG]
    [​IMG]
     
    • Like Like x 1
    • LOL LOL x 1
    • Love Love x 1
Thread Status:
Not open for further replies.