QualifEye - MTurk Qualification Watch System

Discussion in 'News & Announcements' started by ChrisTurk, Feb 28, 2019.

  1. ChrisTurk

    ChrisTurk Administrator

    Just copying this into it's own thread since it'll be lost to the sands of time otherwise in the Daily subforum.

    QualifEye? What is this? Why do you love puns?

    The long & short of it is that you can now use TurkerView itself to alert you to brand new qualification tests & potential qualification hits on MTurk. You don't need to be logged into MTurk, and this will not use any of your valuable page requests to do it's work.

    This allows you to focus on the work actually available to you, while opening up new opportunities during down time and eliminating wasted time & focus. You can actually utilize the "qualified only" options available in most scrapers to filter out the massive amount of junk that you will never be able to do. That is one of the few filters that applies directly to MTurk which actually increases the coverage of HITs your scraper will actually get to scrape & see.

    This will also hopefully help good requesters get their work done. In many cases requesters who go through the trouble of setting up qual tests just have their HITs languish in the void because of how difficult it is to find & complete them. So we're looking to bring good workers together with those kinds of HITs & make it easier for both parties.

    I think the name is funny, I'll change it some time in the future I'm sure since it's objectively pretty dumb lol. QualiFyre, maybe? ..wait I just typed that out and it sounds amazing.. "qualifier" => qualifyre.. who is coming to the festival? :emoji_joy: Sorry y'all, you know I get delirious near feature launches by now!

    Current Features:
    • Tracking & auto updating of qualification tests on MTurk
    • Tracking & auto updating of HITs that are likely qualification, qualifiers, demographic, or pre-qualifiers for batches/surveys
    • Tab alerts
    • Live desktop push notifications

    Qual Tests

    QualifEye has a few thousand qualification tests indexed already (though for the moment it will only display the last few days worth), and it picks up new ones to the platform (or old ones making a return to being used) automatically.

    In the last day of testing both of these HIT groups came about from me using QualifEye (the first had 100 available, the other 300, and both sat around for hours):



    They added up to $44 worth of HITs I didn't see on scraper but did pick up the qual test & did the batches from off QualifEye and I only caught them while testing/building it, not actively using the tool. Of course YMMV, but I think QualifEye can potentially add a lot of value to your workflow & am excited to see how it's utilized!

    Qual HITs

    In the past few days it picked up GeoHives, Cloud Labor, & a Unanimous AI qual along with dozens of other small screening/demographic surveys. I'll be improving it based on feedback from this pilot run over the weekend, of course, but it's already pretty good at what it does.

    As with all things I expect some bugs & glitches that I'll need to work out over the weekend, but I'd like folks to be able to get a feel for it & if it has actual value to them so the plan is to knock out any user/browser specific bugs by Monday :)

    If you've got questions, comments, feedback, & the normal gauntlet of words I use here to ask for your opinions as always just throw 'em at me. I need a nap so will be back around in the morning.

    Where Is It!?
    You can access it directly through this link: https://turkerview.com/qualifeye/

    QualifEye Link

    Aaaand as always thank you to everyone who gave suggestions & helped me test things out! :emoji_heart_eyes:
    • Love Love x 7
    • Like Like x 4
    • WOW WOW x 1
    • Useful / Informative Useful / Informative x 1
  2. LAM

    LAM New Turker


    I have a minor issue with QualifEye.

    A pair of weeks ago I had requested a feature by email: to show the requirements for each Qual Test.
    It was important for me because as a non-US worker I don't qualify for most HITS and I can't find it out immediately.
    Kindly you took care of it adding the "(i)" control, specifically intended to show such requirements.

    Alas, at least for me, it works just seldom.
    Usually clicking the (i) the next Qual Test just slides out of view like to make space for something else but no "qualifications required" appears.
    Sometimes, rarely, it works but just for the first Qual Test listed.

    Using Chrome 73.0.3683.86
    Same issue with Firefox 66.02

    • Like Like x 2
  3. Dawnia

    Dawnia Turker

    @ChrisTurk is there a way to filter out HITs that require masters?
    • Like Like x 1
  4. EsaEinai

    EsaEinai New Turker

    It's been a year since I started on Mturk, and I wanted to say a huge THANK YOU for making this Qualifeye. On Mturk, I found the qualification tests hard to find, which were hidden in a small drop-down underneath each HIT listed on the original queue on mturk (right where "request qualification" might be listed). It was cumbersome and time consuming, and I barely found any. This Qualifeye has helped me immensely, not only finding those tests and consent forms, but also the possible Qualification HITS. It has really helped me in my work. Thank you so very much!!!
    • Like Like x 2