Help: Pay Rate for Extra Long HIT?

Hi everyone,

I was hoping you could all give me feedback about the appropriate pay rate for an upcoming HIT that will take about 60 minutes to complete. In particular, I'm interested in what personal goals you all set for yourselves in terms of effective hourly pay rates on an average day.

The reason I ask is that some people will need to complete my HIT as a group, and I'd really like to make sure everyone moves through the HIT together at the same pace, so that no one's time is wasted waiting for teammates who may occasionally leave the HIT to work on other tasks. I figure if I can guarantee that everyone will meet their personal earnings goals for the hour they participate in my experiment, that might make people more willing to really commit to 100% focus on the HIT for the full 60 minutes.

Generally, I set a base fee equal to the Federal minimum wage ($7.25/hr), and offer a performance-based bonus that averages around $5.00/hr. So, with the bonus, the average person will receive a base fee + bonus that's about equal to the California minimum wage ($13.00/hr).

It seems like those of you who've completed my recent HITs have been pretty satisfied with the effective pay rates I'm using. But, I also know that a lot of you really value control over the way you allocate your time, and manage your task flow throughout the day. So, I wonder if it's necessary to increase the pay rate to compensate for the extra burden of requiring 100% focus, with no side tasks or distractions, for such a long period of time.

(For context: I'm already really pushing the bounds of what my department/advisor will approve in terms of per-person pay rates, so I'm trying to figure out if it's even possible for me to push the compensation high enough to get the focused participation we need.)

And, just to make sure Chris knows I'm not ignoring his blog posts, I should mention that I have reviewed the wage sentiment analysis (https://blog.turkerview.com/2019-sentiment-analysis-mturk-wages-measuring-fairness/), but what I'm trying to figure out is whether I need to provide a sort of "inconvenience bonus" when running very long tasks during which people do not have the option to set their own pace, or leave and return at various points during the task.

For those of you who have completed The Sales Game (my most recent HIT), I am particularly interested in your thoughts on this. The upcoming HIT will be very similar, but might take about 10 minutes longer to complete, and some people will need to complete the HIT cooperatively with teammates.

 

Disclaimer: the inconvenience in my book would be dealing with other workers. Coop tasks have a high stink rate.

Either way, an alternative school of thought that I don't particularly agree with for my own time but some people find that a guaranteed floor of a block of time completely squared away & paid for by a large HIT to be valuable in and of itself.

SelfConcept is a good example of this.. they take something like 10 hours to complete, the base HIT reward is $60 and then they bonus workers up to $10/hr for the rest (maybe? I think. It used to be 7 [$60+$10 bonus] and that was actually yuck). I will never do them, but a lot of workers appreciate them because it's a large guaranteed payoff. Hard disagree with it as a concept, but I don't make the rules for everyone's lives so there are people who are fine with it.

The caveat there being SC can be completed across multiple days and the timers are huge, so people usually save them for a weekend (when their effective hourly would be hugely constrained by a lack of work) and just pick at the HIT in bits/pieces to salvage their sanity.

All that to say there is never any perfect solution for everyone on a platform like this. I think the ~15/hr mark is fairish if you're working along the lines of budgetary constraints and such, but I'd dip out on the basis of it being a cooperative task lol. Anecdotal, but I can usually multi-task tasks like that for a higher effective hourly simply because it is all but a guarantee a group gets at least one person who feels the need to contemplate the true meaning of the universe for every simplistic question the group has to deal with

Not advocating for a lower rate, but IME of coop tasks the people aiming for perfected hourlies aren't the slowpoke/multi-tasking (in a problematic way) culprits.. its the folks who have their dog/kid decide they have to go out right now and AFK it, or someone wrestling with assistive technologies on bad UI/interface, etc that holds up the group. Highly optimized workflow workers are going to have the main task on their focused monitor, and filler work to the side for if/when they'd otherwise be facedesking their keyboard.

Most workers wont selfishly just dip on a task that is progressing productively for no reason, the filler work is only there to save them from, idk, things like punching their monitors in frustrating situations. Caveats there include things like multi-acceptable work (where you can quad+ screen them) and straight bailing for immensely overpaid work that happens to post mid-task. Doubt the former applies, not many constructive ways to combat the latter.

I mostly came to spam the gif, but if you let me know the title of the HITs you were talking about I can tag workers who've done them and have them take a look at this since it seems like you were looking for feedback from them specifically? Otherwise sorry, you know I love to ramble haha. Will post this Mon/Tue to the daily to try and get you some more eyeballs

 

Yes, I completely understand why multiplayer tasks are the bane of everyone's existence. Believe me, every time I run one of these, I suffer as well. But, at least you guys have the option to avoid them. I study group decision-making the emergence of social networks, so I will be stuck running multi-person experiments for the rest of time. Pray for me.

The title of the recent HITs is "45-minute game about learning and searching for solutions." These were all single-player versions, so anyone who's done these and wants to weigh in, keep in mind when you're thinking about a fair rate for this new task that the new version will take about 10 minutes longer to complete, and most people will need to complete this new version as part of a group.

Also, those of you who completed the old HIT saw that almost every page had a time limit, which I agree is very stressful, but I have to include limits to make sure that everyone takes about the same amount of time to make each decision throughout the task - both because a) longer decision times may facilitate different psychological processes, and we need to keep everyone on the same "mental" track, and b) when we move to multi-player, I don't want anyone getting stuck on a page waiting for that one guy contemplating the true meaning of the universe.

I noticed that on the few pages of the old HIT that did not have time limits, a lot of people used those as an opportunity to take a 5-10 minute break. This is of course totally fine for single-players, but will obviously cause problems when we move to multi-player, so the new version will have time limits on every single page. (There's also a pop-up that reminds people to stay on the task screen, which @Azazael gave me some good feedback about, but which we probably have to continue using, because we just get way too many people letting the time run out without submitting any response over and over again in the versions we tried without the pop-ups.)

So, given that we're imposing time limits and auto-advancing everyone through the task, there *shouldn't* be any significant delays once the task gets started. There may be a 2-5 minute wait for the group to assemble at the beginning of the task, but I build the maximum wait time into the estimated time for completion to make sure everyone is paid for waiting as well.

The real problem is that if one person on the team does get up and leave their computer, then when the task auto-advances, we won't record any response for that team member who got up and left. This causes some serious problems for both the other team members, and for me, which I can't describe in detail without biasing folks who might take part in this new experiment (but note that an absentee team member should not adversely impact anyone's chance to maximize their own bonus payout, it just changes the nature of the game). The point is that I just need to minimize the chances that we'll have a wanderer who ruins the experience for everyone, and I'm trying to pull every lever I can to do that, including pay rate.

If anyone's had experience with other multi-player HITs where they think the Requester had a good (non-monetary) method for screening people and setting up groups (e.g. pre-screening questionnaire, scheduling game times in advance), please feel free to share those methods as well. I just want to know about any/everything I can possibly do to make this multi-player experience not terrible.

 

Wait, what? I did nothin....oh yeah, I helped. And yeah we did discuss a lot of things about the survey/hit but it really can't be "fixed" given the nature of what the testing to real product will be. Overall the multiplayer game system is both the bane and the fun of mturk (imo). Sometimes you get great working programs like a few of the requesters, and then you get ones that are just...broken. You get there and sit and wait, no one shows up. You show up, people join, leave, you get no bonus. There is just so many issues with multiplayer games and sadly you never know what you will eventually get until you get it.

 

Except for remesh. And then you know exactly what you're getting but do it anyway

Ohhhh I LOVE this thread: https://forum.turkerview.com/threads/launching-multiplayer-games-poor-click-through.2419/

He was incredibly kind to share the data on the thread. I have it bookmarked & reference fairly often.

@Abdullah if he's still around would definitely be someone I'd reach out to, if not some of his older threads (this subforum) might have something useful.

These I'm not familiar with & don't think they're still around but they have a lot of threads so might have something to parse down: https://forum.turkerview.com/members/multiplayergames.15933/