[Please Help!] How to get 36 people online at the same time (multiplayer game)

Hi everyone,

I am a noob requester and I would like to get some feedback and help on running an online multiplayer game on MTurk. The game requires 36 people to be online simultaneously (players are randomized across across different conditions).

So far, I tried to notify a panel of turkers and post on forums about the upcoming experiment time. I then posted a batch of 45 HITs with a link to our platform. In this HIT, the base pay was $2 and one can earn up to $8 of bonuses based on performance. The bonus varied across conditions, as some conditions had higher average play time.

The first 36 workers who clicked the link and passed the attention checks played the game. For the surplus workers (9 workers), they received the base pay of $2 and went to the exit survey directly. I also gave the workers the option to leave the lobby after 3 minutes in case the game didn't start (they also received the base pay of $2 should they decide to leave or they get the option to remain in the lobby and continue waiting).

This hasn't been a huge success the last time I tried it. It took 30 minutes for the game to accumulate enough players to start. As many people left after waiting for 3 minutes in the lobby. I also had to submit another batch of 20 HITs to keep pumping people in.

I am wondering if there is a better way to do this. One way I've been thinking about is the following:
Having a batch of 100 HITs with $0.1 base pay. The first 36 players will play and receive the base payment of $2 + the bonuses based on their condition. The remaining 64 players (surplus as the game already started, or if they left the lobby) will receive $0.1 .. This way, I can keep pumping players into the lobby until the game starts (so people will not have to wait long) and at the same time, I keep within the budget of the project.

Does this sound like a fair thing to do? I am open to suggestions and guidance.

 

What seems to work is to have a qual hit and then tell workers to show up at X time. Or if you get the qual hit out to enough people you can have them pick times to show up or w/e. Group hits on mturk tend to be some of the hardest things for requesters to do well though.

@JonAtwell Does a ton of group hits so he might be able to point you in the right direction for stuff although his groups are much smaller.

Pumping in new players could work.. but for your data it might be better to just have a qual hit that then tells people to show up later.

 

UNU
contact @MyLegIsFine

 

Like Randomacts said, do a qual hit, but for more players than you need a day or two in advance, and give us the date, name of the hit and start time. Some Requesters send us a reminder email and ask us to verify we will be available at that time and will play. Then if someone does not respond, you could go to the next qualified player on the list that wasn't in the first 36 and invite them.

 

Thanks @Sunlite @NBadger @Girl Polar Bear @Randomacts for your advice. I think this sounds reasonable and I can try for our next run of the experiment (each run has 36 players, and we will be running the experiment 10 times -- total of 360 participants). However, I am not sure what qual hit means. Is this a special type of a hit that mturk provide as an option, or is it a regular hit (with no pay) that notifies takers about the starting time of the actual HIT?

 

Qual or qualification hits are hits that are put up to screen people for the demographics ect that a requester wants.

It might help you to read our wiki page here: https://turkerhub.com/wiki/Qualifications

from the wiki

 

Great thanks! we will investigate this way of running the experiment next week. As for this week, everything setup to get people in the experiment with an improved version of the old way.

 

@Abdullah

I run large group experiments (25+... @Randomacts ). I built up a "subject pool" by first posting lots of non-group HITs that trained peeps. Because they've completed one of my HITs, I can email them directly at anytime. The night before I post a HIT, I let them know they're invited to a HIT the next day. Then 30 minutes before I post it, I remind them. Finally, as I post the HIT, I send out a link with the HIT_group link. Currently it takes around 2-3 minutes to fill the spots, except on Friday which takes longer.

The key to making this work is to invite more than you need (roughly 3x, see attached paper) and use qualifications. I assigned everyone in my subject pool the same qualification and adjust the associated score in order to control who can play on any given day. That way you don't have the same, fast-fingered people every time.

It took me a lot of trial and error to get up and running...good luck!

 

I must have been understanding how your hit works incorrectly. I was under the impression that it was a a lot of smaller groups into a meta large group.

 

You aren't wrong really...you interact with the same small group the whole time, but those people aren't necessarily interacting with the same people you are.

Anyway, the instructions don't make that clear because it shouldn't matter. (Plenty of people still think they're playing the same person the whole time!)

 

@Randomacts

Thanks for all your help. We will actually run the experiment tomorrow (Thursday) using the approach you specified. Both, submitting a qualification HIT 1 hour before the experiment. Also, assigning qualification for people who already played the HIT to be excluded from the experiment that they can't play.

@JonAtwell Great! thanks for your help. This has been very useful for us! see you at the Collective Intelligence Conference in NYC this summer!

 

Did you solve these problems somehow?