Trouble with having our HITs completed within the time frame

Discussion in 'Requesters' started by Robert, Jul 13, 2018.

  1. Robert

    Robert Hummingbird Watch

    Messages:
    158
    Gender:
    Male
    Ratings:
    +310
    Ugh. Trouble in paradise.

    We are maybe having one or two HITs completed in each batch. Which is not enough. We think they are expiring in peoples queues before they get to them. With wildfire detection, of course, its important that its done in a timely matter.

    We tried to increase the expiry but decrease the time limit, no luck. Anyways, we thought we would put it out to you guys in case you had any suggestions.
     
  2. Tripsa

    Tripsa ┴ɹᴉdsɐ

    Messages:
    11,895
    Gender:
    Male
    Ratings:
    +30,588
    Might be time to go the qual route on the hits if people are catching them just to let them expire in their queue's. Hits will then go to a group of turkers who are known to submit them when they are able to catch them.
     
    • Like Like x 2
  3. ChrisTurk

    ChrisTurk Administrator

    Messages:
    56,679
    Ratings:
    +162,927
    If its not proprietary, seeing the settings you're using might be helpful. Specifically:

    HIT Duration
    Time to expire
    Quals attached

    Also can someone send me his req id? Idk any of the main info anymore haha :flail:
     
  4. Robert

    Robert Hummingbird Watch

    Messages:
    158
    Gender:
    Male
    Ratings:
    +310
    So we would just increase the qualification requirements?
     
  5. Randomacts

    Randomacts Survey Slinger

    Messages:
    94,622
    Gender:
    Male
    Ratings:
    +124,073
    Yeah I was doing them but I got annoyed with the catch rates for having a panda up all the time so I ended up turning my panda off yesterday.
     
    • Like Like x 1
  6. queenpenguin

    queenpenguin Well-Known Turker

    Messages:
    1,770
    Ratings:
    +2,523
    Checked my HIT Tracker with you. I did 48 for you and none of it run out of time for me. No problem for me with 5 minutes timer.
     
    • Like Like x 3
  7. Randomacts

    Randomacts Survey Slinger

    Messages:
    94,622
    Gender:
    Male
    Ratings:
    +124,073
    Title: Hummingbird Watch - Early Forest Fire Detection | PANDA
    Requester: Robert Atwood [A2AARFMJHG94SR]
    TurkerView: [ $10.33 / hour ]
    Description: Thank you for your interest in this HIT! Here are the instructions:

    Inspect each image for smoke, if you find some click the flame button at the bottom of the image. *Remember, smoke can be hard to detect! Look closely.
    Once you have selected that your image contains smoke you will be asked to draw a bounding box around the smoke. Once you are happy with the area you have selected, press the green check button.
    If no image contains smoke or flames, click the green forest button at the bottom of the page. You will automatically be directed to the next image.
    Once you've looked through all images, the HIT will be automatically submitted.
    Here are a couple of tips:

    We are testing for speed and accuracy, so please try and complete this task as quickly as possible. We are aiming for a submission time of 1 - 2 minutes with practice.
    Most of the time there won't be any smoke, so don't doubt yourself if you can't find any!
    If you think you see smoke, but aren't sure - make sure you mark it anyway. We don't mind mistakes if someone thinks they see smoke. What we don't want is people missing smoke when there is some.
    We are documenting any and all feedback on our HITs, and will be adjusting our HITs accordingly. If you have any feedback or comments, feel free to post on our threads at http://www.turkerhub.com
    Duration: 00:04:00
    Available: 1
    Reward: $0.09
    Qualifications:
    • Location In CA, US
    [3WNTBWVI6HYMS2VB8BKP75ZKBFD1DV]
     
    • Today I Learned Today I Learned x 1
  8. Tripsa

    Tripsa ┴ɹᴉdsɐ

    Messages:
    11,895
    Gender:
    Male
    Ratings:
    +30,588
    This. I turned mine back on today and have only caught eleven so far in the past four hours.
     
    • Like Like x 1
  9. Robert

    Robert Hummingbird Watch

    Messages:
    158
    Gender:
    Male
    Ratings:
    +310

    MaxAssignments : 6,
    LifetimeInSeconds : 600,
    AssignmentDurationInSeconds: 240,
    Reward : '0.09',
    QualificationRequirements : [{
    QualificationTypeId: '00000000000000000071',
    Comparator : 'In',
    LocaleValues : [{Country: 'US'}, {Country: 'CA'}]
    }]
     
    • Today I Learned Today I Learned x 1
  10. ChrisTurk

    ChrisTurk Administrator

    Messages:
    56,679
    Ratings:
    +162,927
    How many HITs are you releasing per hour (ballpark is fine)? Is it still a set # of hours/day (ie, is there a semi-consistent schedule)?
     
  11. Robert

    Robert Hummingbird Watch

    Messages:
    158
    Gender:
    Male
    Ratings:
    +310
    What does this mean, exactly @Randomacts ?

    Does this mean you turned off the automated script that catches the HITs because you weren't getting any?
     
  12. Robert

    Robert Hummingbird Watch

    Messages:
    158
    Gender:
    Male
    Ratings:
    +310
    Yea, super consistent schedule. Right now we are running a batch every 10 minutes. This will be increasing as the summer goes on and we increase the cameras to the system, but for now
     
    • Today I Learned Today I Learned x 2
  13. Robert

    Robert Hummingbird Watch

    Messages:
    158
    Gender:
    Male
    Ratings:
    +310
    Right, this doesn't make sense to me. Or it does, but its frustrating. You should be getting more.
     
  14. Tripsa

    Tripsa ┴ɹᴉdsɐ

    Messages:
    11,895
    Gender:
    Male
    Ratings:
    +30,588
    Yeah, need a script to try to catch these on regular basis.
     
  15. Randomacts

    Randomacts Survey Slinger

    Messages:
    94,622
    Gender:
    Male
    Ratings:
    +124,073
    Basically yeah. I was getting hits but it still used my limited resource pool of page requests for a 9 cent hit so I decided to just turn it off.

    And we need to leave it running so that means people will walk away from their computer with it running still and hopefully catch the hit before it expires ect.

    Yeah :dunno: too many people fighting over them but actually doing the hits?
     
    • Like Like x 1
  16. Tripsa

    Tripsa ┴ɹᴉdsɐ

    Messages:
    11,895
    Gender:
    Male
    Ratings:
    +30,588
    Most turkers on here use a script that allows them set what hits to try to catch. we call it panda. Find more info about it here. https://turkerhub.com/wiki/panda#Legacy-mTurk-com

    I turned mine back on this morning with it trying to accept your hits.Today has not been a good day trying to catch these since others are trying to catch them as well. Think I got up to around 60 hits yesterday. I tend to leave it on all day unless some other hits drop that I can catch easier so I'll switch off the panda for your hit to work on others. Then come back to the fire hits once I have caught up with my queue.


    Please forgive my wacky typing I need something with caffeine in it.
     
    Last edited: Jul 13, 2018
  17. ChrisTurk

    ChrisTurk Administrator

    Messages:
    56,679
    Ratings:
    +162,927
    Awesome, thanks.

    So yes, some people aren't trying to catch them because the # they catch/hour isn't worth the "cost" - on mTurk you can only try to catch so many things at once (most people keep it down to 2 or 3 things at any given time) so fitting your HITs into rotation isn't "worth" it to them right now.

    What others have suggested is that by decreasing the # of workers who can catch them you might increase your completion rate.

    That is.. a kinda/sorta dangerous game to play IMO at a maximum rate of 54c/hr (1 hit per 10min available @ 9c each). If you can get people here to commit to that, great, I would say for every 1-2 people you "closed qual" split your HITs up into 2 different groups.

    Public Group -> The ones you're running now, so you can still recruit new/potential workers + "advertise" as it is so others pick you up
    Closed Group -> Ones you know will get done by "good" workers, who are around reliably

    But I think you're still gonna struggle a bit to maximize completion %s when you only have 50c/hour worth of work to do given the extremely tight timing window. I think a simpler first step is:

    Overbid
    With this method, if you need 6 HITs done you should probably release 10-12 (adjust it so that you settle on a completion percentage that nets you around 6 submitted HITs per "release". You're going to "waste" some money from time to time when a particular HIT gets completed more often than you'd expect, but you'll hit your completion targets which is the more important priority (I'm assuming).

    Lets say right now you're averaging 3 HITs completed with 6 HITs released (and you want 6 done). If you release 12 HITs, and your completion [before expiration] % remains at 50%, you get 6 HITs completed [target realized] - only pay for 6 HITs completed [budget on track] and you don't have to futz w/ quals and coordinating workers etc etc.

    The downside is that occasionally you'll have a batch where 9, or even all 12 HITs are completed & you are over budget. You can adjust payment to account for this, or just eat the cost.

    A more long-term solution if you really want pinpoint accuracy:

    Row Your Own Boat
    Building out your own platform solution. I only suggest this because its essentially already done since you aren't hosting the HITs on mTurk's default HIT templates. So find a quick push notification library for a browser, set up a page where an authorized worker can "log in" and complete work as soon as it comes up.

    Just bonus them on mTurk to make your life easier.

    This way, you can have folks you already have a working relationship with just dedicate a small amount of screen real estate to you (just the size of a browser push notification & a browser tab somewhere that doesn't even have to be visible) and as soon as you have work up they can alt/tab to it, do the work, and you just pay them a lump sum EOD/EOW.

    This is a bigger development commitment, but it scales way better than mTurk does without all the pitfalls you're hitting now because of the unique aspects of your work.

    A horrible, totally not recommended solution is:

    The Pinterest Method
    Instead of releasing 6 hits, you can release 10-20, but on your back end server set it up so that only 6 unique workers can "see" the HITs to be able to submit them at any given time, once 6 have submitted the rest will just say "sorry no work available"

    I strongly recommend against this, as you do not have the budget to piss off workers like Pinterest does and still get away with it. But this method does keep you fully on mTurk w/o having to roll your own solution.

    This is essentially what I suggested above, but you eliminate the "overpay" possibility (at the expense of really pissing off workers, who will report your account to mTurk because this is technically against the TOS)
     
    • Like Like x 3
    • Today I Learned Today I Learned x 1
  18. Randomacts

    Randomacts Survey Slinger

    Messages:
    94,622
    Gender:
    Male
    Ratings:
    +124,073
    There is what is basically an anime version of Osmosis Jones airing right now lol.




    Don't worry it is SFW lol.
     
  19. ChrisTurk

    ChrisTurk Administrator

    Messages:
    56,679
    Ratings:
    +162,927
    @Robert can't stress this one enough, its the actual simple solution to your current problem (and the one most reqs in your position turn to first), I just laid out all the rest as options lol.
     
    • Like Like x 1
  20. Havoc928

    Havoc928 Active Turker

    Messages:
    854
    Gender:
    Male
    Ratings:
    +818
    • Like Like x 2