Planning to drastically reduce assignment duration for UserBob HITs

Lately I have been having a harder time getting HITs completed in a timely fashion. I can tell that HITs are getting accepted but not actually worked on. My assumption is that workers are using scripts to automatically accept a HIT but then the worker is not available to actually complete the HIT. While the HIT is sitting in this queue, no one else has the opportunity to work on it until it expires and is made available to another user. This appears to repeat several times before the HIT gets accepted by a worker who is actually going to work on the HIT.

Here's what I'm thinking of doing to address this problem. Instead of having a HIT that says a worker has an hour or more to submit it, I will create my HITs to only give users about five minutes to submit the HIT. Now, I know that some of my HITs can take longer than five minutes to complete. For example, if the HIT asks you to review a site for 20 minutes, there is no way that you can finish that task in five minutes. Instead I will be asking you to accept and review my HITs and then submit them before actually doing the associated work. As long as you upload your video in a timely fashion everything should turn out fine.

Please let me know how you'd feel about such a change.

 

Just at first glance I feel like this is likely to cause as many headaches as it'll solve. Probably more honestly since some workers will just space case not going back to finish/do the HIT and then they're already paid & a fair number will also go anxiety mode on this kind of setup (what if their power goes out? internet goes down? bitten by a shark? etc). Either way I'd expect an increase in required human interaction that'd be kinda difficult to automate/solve around.

Would maybe suggest to just use the notify workers api to send a warning message to repeat offenders who are causing problems from this, then de-qual them if they don't stop doing it. Both of those should be more approachable from an automation standpoint. AFAIK there aren't any auto-accept tools that will go search for re-uploaded HITs so I'm not sure how much of that is actually that kind of thing, but either way decent non-abusive ones don't let users easily exhibit this kind of behavior because engaged devs know this kind of thing causes problems. I could be wrong here, but if the distribution is across a low overall portion of your workerbase I'd lean more towards it not being a widespread tooling thing. Kinda just a blind guess though.

It's an understandable thing for a worker to do every once in a while (things come up & you gotta bail) but if it's noticeable on your end then yeah, this probably isn't normal behavior.

Just my 2c, I don't do those HITs so probably not as valuable as direct feedback. Of workers I can recall seeing review your HITs a lot: @Rin @Troy @NBadger @TheNerdyAnarchist @Sparrowmint might be better to ask so tagging them for ya.

 

I wonder if this will cause more problems than the goodwill earned from your customers is worth.

Particularly for lower level reviewers.

I know the last ~20 minute UB HIT I did had like a 2 hour timer on it which is a bit much tbh.

Why not just use ~35 minutes for all HITs or just use something like HIT time + 5-10 minutes to determine timer length?

 

would also make me curious if this is a problem occurring across lvl 3-4 HITs (where workers here will know better, or care that it causes you problems) or lvl 1-2 HITs or both.

 

I've done a couple of your hits in the past. I really like them but I have had to return one or two after accepting and doing them because of internet issues on my end. It just takes too long to upload for me sometimes and it overheats my laptop. It might be that some users are just having technical issues on their end like I have. Personally, I just don't accept the hits any more because I know that I'll delay completion if it takes me forever to upload.

I agree with @A6_Foul_Out that shorter timers could work better because they will be prioritized in the queue by most workers.

 

Thanks for all the feedback. The problem is mostly only a problem with the longest HITs. I'll start out with a smaller adjustment to the timeout and see if that helps.

 

my upload is about 3mbps and this change would severely impact me. I use scripts (as basically everyone else) but i work them immediately. You have always said that if the hit expires we can email and you will approve, but that would cause a lot of manual work from both your and our end. I would suggest to only apply this change to level 1 workers. the recent 25 minute hits i was lucky enough to be able to use phone data (which is limited for me) to upload it quickly. If i had used my wifi connection it would have taken me nearly 1 hour to upload

 

@UserBob

Sorry didn't see this sooner, been busy with the start of the school year as a teacher.

I know I've done over 200 submitted videos at this point for you (no rejections), including many that are 15-25+ minutes, and this would make me hesitate to try for/accept HITs by you. Just saying.

I do occasionally return HITs by you because of technical issues on the end of whatever I'm reviewing (sometimes I will retry and it will force me to return it), or I run into some aspect of the site/app midway that skeeves me out because of personal information appearing onscreen or whatever.

But I try to always be conscientious with my reviews and my effort into them. Ultimately though, if I run into a situation where I think what has happened won't meet the required terms of the review, then I'm not going to submit it and risk the rejection. So yeah, if I'm having to submit blindly and then just hope that the review is possible to do, just not worth the risk to me.

 

Hopefully the lower times have helped, on ~20 minute videos the timers have been ~60 minutes?

Could go down to 40 if further improvements to turnaround time needed to be made.