[Please Help!] How to get 36 people online at the same time (multiplayer game)

Discussion in 'Help & Guides' started by Abdullah, Apr 9, 2017.

  1. Abdullah

    Abdullah AGEIRIHRRYV6P

    Messages:
    321
    Ratings:
    +630
    Hi everyone,

    I am a noob requester and I would like to get some feedback and help on running an online multiplayer game on MTurk. The game requires 36 people to be online simultaneously (players are randomized across across different conditions).

    So far, I tried to notify a panel of turkers and post on forums about the upcoming experiment time. I then posted a batch of 45 HITs with a link to our platform. In this HIT, the base pay was $2 and one can earn up to $8 of bonuses based on performance. The bonus varied across conditions, as some conditions had higher average play time.

    The first 36 workers who clicked the link and passed the attention checks played the game. For the surplus workers (9 workers), they received the base pay of $2 and went to the exit survey directly. I also gave the workers the option to leave the lobby after 3 minutes in case the game didn't start (they also received the base pay of $2 should they decide to leave or they get the option to remain in the lobby and continue waiting).

    This hasn't been a huge success the last time I tried it. It took 30 minutes for the game to accumulate enough players to start. As many people left after waiting for 3 minutes in the lobby. I also had to submit another batch of 20 HITs to keep pumping people in.

    I am wondering if there is a better way to do this. One way I've been thinking about is the following:
    Having a batch of 100 HITs with $0.1 base pay. The first 36 players will play and receive the base payment of $2 + the bonuses based on their condition. The remaining 64 players (surplus as the game already started, or if they left the lobby) will receive $0.1 .. This way, I can keep pumping players into the lobby until the game starts (so people will not have to wait long) and at the same time, I keep within the budget of the project.

    Does this sound like a fair thing to do? I am open to suggestions and guidance.
     
    • Like Like x 2
  2. Randomacts

    Randomacts Survey Slinger

    Messages:
    94,622
    Gender:
    Male
    Ratings:
    +124,073
    What seems to work is to have a qual hit and then tell workers to show up at X time. Or if you get the qual hit out to enough people you can have them pick times to show up or w/e. Group hits on mturk tend to be some of the hardest things for requesters to do well though.

    @JonAtwell Does a ton of group hits so he might be able to point you in the right direction for stuff although his groups are much smaller.

    Pumping in new players could work.. but for your data it might be better to just have a qual hit that then tells people to show up later.
     
    • Like Like x 2
  3. Girl Polar Bear

    Girl Polar Bear Queen of the North

    Messages:
    29,273
    Gender:
    Female
    Ratings:
    +45,742
    UNU
    contact @MyLegIsFine
     
    • Like Like x 1
  4. NBadger

    NBadger Mod of Cat Gifs

    Messages:
    14,376
    Gender:
    Female
    Ratings:
    +40,241
    Qual HITs are the way to go,* or at least having a set start time. UNU (@MyLegIsFine) does a good job of sending out HITs a couple hours before the group start time—e.g., the HIT is released at 1pm for workers to accept and then the HIT itself starts around 3pm (well, when mturk cooperates anyway) and that tends to work pretty well.

    Yale group HITs also tend to work on timers, so when you accept the HIT it tells you the game will start in X many minutes to make sure everyone has time to get in/starts at the same spot.


    *Feel free to award me with said quals** for my Excellent Advice Skillz™

    **jk. But also not jk. But also jk :ag:
     
    • Like Like x 1
    • LOL LOL x 1
  5. Sunlite

    Sunlite Survey Slinger

    Messages:
    5,028
    Gender:
    Female
    Ratings:
    +7,755
    Like Randomacts said, do a qual hit, but for more players than you need a day or two in advance, and give us the date, name of the hit and start time. Some Requesters send us a reminder email and ask us to verify we will be available at that time and will play. Then if someone does not respond, you could go to the next qualified player on the list that wasn't in the first 36 and invite them.
     
    • Like Like x 1
  6. Abdullah

    Abdullah AGEIRIHRRYV6P

    Messages:
    321
    Ratings:
    +630
    Thanks @Sunlite @NBadger @Girl Polar Bear @Randomacts for your advice. I think this sounds reasonable and I can try for our next run of the experiment (each run has 36 players, and we will be running the experiment 10 times -- total of 360 participants). However, I am not sure what qual hit means. Is this a special type of a hit that mturk provide as an option, or is it a regular hit (with no pay) that notifies takers about the starting time of the actual HIT?
     
  7. Randomacts

    Randomacts Survey Slinger

    Messages:
    94,622
    Gender:
    Male
    Ratings:
    +124,073
    Qual or qualification hits are hits that are put up to screen people for the demographics ect that a requester wants.

    It might help you to read our wiki page here: https://turkerhub.com/wiki/Qualifications

    from the wiki

     
    • Today I Learned Today I Learned x 1
  8. Abdullah

    Abdullah AGEIRIHRRYV6P

    Messages:
    321
    Ratings:
    +630
    Great thanks! we will investigate this way of running the experiment next week. As for this week, everything setup to get people in the experiment with an improved version of the old way.
     
    • Like Like x 2
  9. JonAtwell

    JonAtwell New Turker

    Messages:
    3
    Ratings:
    +14
    @Abdullah

    I run large group experiments (25+... @Randomacts :) ). I built up a "subject pool" by first posting lots of non-group HITs that trained peeps. Because they've completed one of my HITs, I can email them directly at anytime. The night before I post a HIT, I let them know they're invited to a HIT the next day. Then 30 minutes before I post it, I remind them. Finally, as I post the HIT, I send out a link with the HIT_group link. Currently it takes around 2-3 minutes to fill the spots, except on Friday which takes longer.

    The key to making this work is to invite more than you need (roughly 3x, see attached paper) and use qualifications. I assigned everyone in my subject pool the same qualification and adjust the associated score in order to control who can play on any given day. That way you don't have the same, fast-fingered people every time.

    It took me a lot of trial and error to get up and running...good luck!
     

    Attached Files:

    • Today I Learned Today I Learned x 1
  10. Randomacts

    Randomacts Survey Slinger

    Messages:
    94,622
    Gender:
    Male
    Ratings:
    +124,073
    I must have been understanding how your hit works incorrectly. I was under the impression that it was a a lot of smaller groups into a meta large group.
     
  11. JonAtwell

    JonAtwell New Turker

    Messages:
    3
    Ratings:
    +14
    You aren't wrong really...you interact with the same small group the whole time, but those people aren't necessarily interacting with the same people you are.

    Anyway, the instructions don't make that clear because it shouldn't matter. (Plenty of people still think they're playing the same person the whole time!)
     
    • Today I Learned Today I Learned x 1
  12. Abdullah

    Abdullah AGEIRIHRRYV6P

    Messages:
    321
    Ratings:
    +630
    @Randomacts

    Thanks for all your help. We will actually run the experiment tomorrow (Thursday) using the approach you specified. Both, submitting a qualification HIT 1 hour before the experiment. Also, assigning qualification for people who already played the HIT to be excluded from the experiment that they can't play.

    @JonAtwell Great! thanks for your help. This has been very useful for us! see you at the Collective Intelligence Conference in NYC this summer!
     
    • Like Like x 2
  13. Mechelsohd

    Mechelsohd New Turker

    Messages:
    1
    Ratings:
    +0
    Did you solve these problems somehow?