Survey on desirability of automated tasks (~10-12min)

Hi! Requester here.

tl;dr Our survey started off getting lots of responses but has slowed down a lot. Please do try it ($2.50, name is "Survey on desirability of automated tasks (~10-12 min)") and if you'd like, let me know what might be causing the slow response rate!

We have a survey asking people to give their opinion on how much they would want a certain task to be automated. I can't link to it as I don't have a worker account, but the name is above. Our response rate started off great and slowed down a ton recently. We would love for you to try it, and would really benefit if you let us know what might be making it undesirable for you/others. Here's what we've tried:

• Removing Masters requirement, just having certain qualifications: helped a lot, but we'd like to go back to Masters if possible
• Changing some wording to clarify what we're looking for
• NOTE: someone suggested changing the interface from the MTurk default. We know it's an annoying interface, but it'd be hard to change the survey format in the middle of the survey. We will definitely do it for future studies. If you think this is a big deterrent even for this study, do let us know.
• NOTE: we don't allow people to do HITs from multiple batches and ask them not to do multiple HITs from the same batch. We know it would be great to be able to do so, but this would create dataset bias (even moreso than we already have) and therefore is an absolute last resort.

Thanks!

 

Sorry - as I said, I posted here at the advice of a worker but if this isn't the right place to post, I can delete and move it.

 

Hi and welcome!

If you're saying that you got a lot of responses immediately when you posted the HIT today and that they've since slowed to a trickle, that's because of how workers go through their queue.

I haven't been able to catch your HIT, though I am watching for it. But basically, we workers try to snap up as much work as we can. This can be hard, and we have scripts that help us with this. Once we grab something, it goes to our queue, where we can have up to 25 tasks set up. MTurk will automatically sort them by when they will expire, which is when they're returned back to the pool for someone else to grab. This is based on the timer that you set when you create the HIT. If something will expire sooner, that's what we'll be given to work on.

Your timer is set to an hour, which is pretty standard. But if anyone has tasks that are expiring in less than an hour, their worker queue will prioritize those tasks first. If they don't get to your task, it goes back to the pool where someone else will grab it, and the cycle will start anew.

If we just accept one task at a time, we would never have enough work to do, since someone else would grab tasks up before we have a chance to get to them. This unfortunately means that the rest of your data might be a bit slow to come in, even to the point where you may have to wait a few hours due to expiration cycles. People have emergencies come up that may cause them to leave their computers, and so in that case, your task would sit in their queue for the full hour before being released for someone else.

 

I would love to participate; however, I am not qualified at 5,600 approved HITs.

 

Speak of the devil -- I just caught it. Since I was busy typing the above reply, my timer is at 14 minutes of the 60 minutes, so my script caught it 14 minutes ago and I am just now going to do it. So there's an example of how that delays things. ;-)

 

@Sun123 There's a few things that might be making it less appealing:

-96% approval rating. If you're struggling with getting new responses, but have a slightly flexible budget, you can approve HITs that don't meet your data standards, but then privately (outside of MTurk) exclude the data from the dataset. This will keep a high approval and attract new people.
You could also post the HIT under a new or institutional name, with 100% approval (Make sure you let people know it's still Sanj, so it doesn't skew your data)


-Personal info collection. If it's not necessary, then removing some of the detailed demographics may make people more comfortable taking the hit.
Is the city necessary? It's much more personal than asking someone their state, etc.


-Weirdly spelled/formatted questions, e.g.
"Do youwant arobot togetyou aliteracola" could be more "legitimate" and "professional" by being spaced correctly as "Do you want a robot to get you a liter of cola?"

------

-There's a limited pool of Masters. Most of them will be respectful and only do the one that you ask. Since the qual is limited, you may be reaching saturation.


-If you want to clarify between hits of different names, you can put a clarifier for that batch in the HIT title:

"THIS IS THE HUNGRY JACK'S KANGAROO BURGER BATCH. YOU MAY ONLY DO ONE HIT FROM THIS BATCH"
"THIS IS THE SWISS CHARD CHOCOLATE BATCH" etc.

With those descriptions, it will be easy for people to identify other hits you may post, to know if they can even do more than one.

 

I completed this HIT with no issues and would complete more in the future if you continue to release them for people without the Masters qualification. This qualification hasn't been given out in years so you are narrowing the pool down quite a bit because workers like myself who have started in the last couple years may never have the chance to get it.

If you are still having problems even with eliminating the Masters qual I am not sure why. I did see one return review on the one I did that said they were unable to type answers in but I did not have that issue.

 

Thanks, this is a helpful insight

 

Thanks, this is all extremely helpful! The spacing issue is a dataset problem - I've been putting off fixing the problematic ones by hand, but it might be time to bite the bullet and do it.

And yes, the rejection rate had been flagged as a concern. That's a great idea re: a new name - we will look into our budget to see if it can handle that.

Appreciate your thoughts!

 

Opps, sorry about that. I should have been a little more specific with instructions about where to post your queries, but it seems to have worked out. Happy to see that you were able to drum up some interest for your survey and get some useful feedback here.