Hi All,

I'm trying to launch a multiplayer game with 20 players. I don't want the players to wait long in the waiting room, so I set a HIT for 60 workers, and ask the workers to begin the task as soon as they accept the HIT. I pay those who complete the training but don't get to play.

The majority of those hits get accepted in less than a minute, but then workers don't seem to click through to the actual game. Those who do click through wait a long time for people who are sitting on the HIT without getting started.

I'm pretty new to this, so I'm not sure if I need to set the time allotted or the HIT expiration to be shorter, or add more than 3x the number I want to play. Any advice?

Thanks!
James

 

I would say password it. Give the group of 20 a code to progress.

 

Sure! This is the HIT that failed to launch:

Play a collaborative mystery game ($1 training + up to $3 to play)(~ 20 minutes) ​

3BMWOOFJUZWL0BOCVSBGE7F8J1MTS3

Its launched through Turkprime, so you end up seeing this:

I ran a game using a qual hit last month, where I had 60 players go through the training, and then spammed them all relentlessly about a game at a specific time the following day. The game launched, but I felt bad about bothering everybody so much. Is that a better route to go down?

 

You're incentivizing this behavior by doing this:

I know it's nice, but it's the wrong way to go about these cooperative tasks unfortunately.

A little of everything.. these game/cooperative tasks are kind of hampered by the way workers accept & prioritize their workflows nowadays. Workers aren't opening your HIT before or even after accepting it, it's just being placed in a queue with up to 24 other HITs and they'll get to it when they get to it. A long timer means it's placed at the back of the list, and many workers wont open it until it is way too late, especially since your HITs don't give any indication that they need to promptly open it (looking at your HIT titles on TV)

Probably the best mix of simplicity + effectiveness:

• Run recruitment HITs that pays 3-5c/ea, have people fill in the time slot available. My honest suggestion to keep costs low here is to simply have a checkbox with "I'm available in 1 hour for a 20 minute game paying $4" or w/e your payment is, come drop by the forum & export the HIT so workers who are around can keep an eye out for it. Republish in batches of 25 until you've got enough workers who have committed to the time that they're basically "penciling you in" to their day
• Give that group a qual (definitely over qual here, it's 5c, grab 100 workers and it only costs $5-$10)
• Publish the HIT 15-30 minutes before the start time with a very clear title that it needs to be opened/checked immediately. Give 10-15 minutes of submission buffer (this needs to be short, as short as you can depending on your confidence workers will not run into issues that delay the game, so that it naturally pushes the HIT higher up their queue workflow by having a lower total duration) from when you expect the task to end.
• Again, most workers aren't really looking at what they accept, they simply look at a list of HITs in their queue and select what to work on based on some combination of hourly pay + duration left on the hit but all of those tools display the title so abuse ALL CAPS and get your point across !!!OPEN IMMEDIATELY!!! Cooperative game $15/hr! *STARTS 3PM EST / 12PM PST* - you need to grab folks attention to the benefit of both parties, it's OK to do this in this instance.
• Stop incentivizing workers to not do your task. Disqual anyone who misses an agreed upon time twice (don't block, just use a disqualification and apply it to all of your future HITs so people get the point).
  • Don't use the recruitment list, track worker ids who accept the task.. if you can't do this (it requires your own external hit), ignore this advice

Some of this isn't the most worker friendly advice, unfortunately many tasks like yours have trained workers how to exploit the system.

 

If this worked for you it's perfectly acceptable.. most workers understand the frustration of trying to coordinate these HITs just as much as you do. Still highly recommend adjusting your titles to also reflect this nagging nature haha :emoji_joy::emoji_joy::emoji_joy:

 

oh, CT got it

 

That explains a whole lot! Thank you!

In the game I ran last month, I had workers go through the whole training (8 mins for the slow ones) before qualifying them, and then scheduled a day out. You're suggesting a more lightweight recruitment task (i.e. commit to the time) with a shorter lead time for the actual game (1 hr), which sounds easier and cheaper.

I've been setting the time allotted per assignment (same as submission buffer?) to an hour. My understanding is that if the time runs out before workers submit the completion code, they don't get paid (am I reading the docs correctly?). The total task takes 3-8 mins for training + lobby time (the failed game lobbied for 15 mins) + 8 mins for gameplay + 2-4 minutes follow-up. If I set the time to 30 mins, will that be a problem? If I launch the HIT 15 mins before the nominal game start, I suppose I should add that 15 mins to the time?

Thanks for the permission. =)

After a $.05 recruitment HIT, I'm thinking about $1 for those who complete the training but don't get to play, and $.10 for those who show up but don't get to do the training (which happens if they show up after the game launches).

Urg... hate being part of the problem...

 

Yes, but you'll likely need to play w/ the lead time a bit, but since it's cheaper failure is less costly (or should be hopefully lol)

That's correct which is why it's generally better to give ample time for most HITs

I think 30 is cutting it too close.. you'd ideally want to launch the HIT 15-30 before it even starts, so something like this:

20 min lead time
15 min training
15 min game
10 min followup (submission should be basically immediately after
​

That's ~an hour duration, which is fairly standard for most HITs so yours shouldn't take forever to cycle towards the upper half of a worker's queue. Coupled w/ them expecting it, the majority should see it that way and you're giving ample time to get everything done.

That sounds fine, maybe I misunderstood I thought workers got $1 just for showing up even w/o doing the training haha.

Haha, I think the problem here is human nature & ingenuity when it comes to finances :p

 

Thought I'd post an update for other folks who may have similar issues.

I took your advice and launched a qualification hit ($.05) called "Qual HIT for collaborative detective game ($1 training + up to $3 play) 3pm EDT Today", just a straight mTurk survey, saying:



This qualified 85 players. Then I set up a HIT for the actual game with 40 slots, called "!!!OPEN QUICK!!! Collaborative game 3pm EDT $4/20mins ", which directed individuals to the game server. I made the HIT visible only to those who had signed up in the qual HIT.

Thinking people would check the HIT at 3pm, as advertised, I launched the HIT at 2:46, just to be ready. The game server was already active, so lots of people went over and got started with training. (Yay!). In the end, there were enough people trained that the game itself launched at 2:57ish. I immediately paused the HIT, so that new people wouldn't start the training and be unable to play (better experience for them, and cheaper). The game server automatically dropped the 6 people who were still in training (less of a good experience, but they got the whole training payment).




Here's a chart of the whole process (Times are UTC). The recruitment was fast at first, but slowed down ~8 minutes in, and then picked up again after at 4:52 when I sent an email to those who had signed up but not joined. (Turkprime makes you wait 6 minutes from when the HIT posts before you can email workers).

The average time from clicking the HIT to entering the completion code is ~20mins, as promised. The only downside here is that by launching the HIT early, there were a number of people who sat in the waiting room longer than they needed to because they got there early.

I think next time, I'll do the same qual HIT, but actually launch the game hit one or two minutes before the advertised time. I think I've also worked out a way to email the whole qualification list with the HIT permalink prior to launch.

 

Thank you so much for posting this. Incredibly helpful as this is almost always a "best guess" kind of recommendation on my part when talking with requesters so I can adjust accordingly now. Can't express how much I appreciate it!

FWIW on the plus side most workers are good about still maximizing lobby times, so sitting in the room is likely just being put into the background while they go off to do another HIT in between the wait time, but awesome to see it filled up that quickly so you can work with narrower windows :)

Just curious, the graph shows a max of ~25 users joining, that is out of the 40 or 85? Post says 85 qualified but the game was "set up" with 40, just making sure I'm clear on how many HITs were posted/available to workers (25/40 is fantastic, 24/85 is a lot lower than I would have guessed)

 

Absolutely! I imagine this sort of data is hard to come by, and I'm glad to share. =)

I have a chime that sounds when the game starts, which brings people's attention back to the game. It seems to work well, as there is moderate activity in the first minute. I'd definitely recommend that to other game devs.

The 3pm (well, 2:45) HIT had 40 slots, so 25/40 is a pretty good ratio. The 85 people who qualified were all invited to that HIT, so there was going to be some people disappointed in not being able to get one.



Thought I'd post a chart of the failed game as well, just as a point of comparison. (This is the case I was describing in the original post). The "Players in Intro/Exit" includes people who accepted the HIT but didn't click through, so you can see the auto-accept and delay behavior pretty clearly.




Here the lobby timed out after 15 mins (from the first player being ready). I'm not sure what I think about having that auto-timeout set. On the one hand, it means that players aren't waiting forever for a game that will never launch. On the other hand, when the lobby timed out, there were still players in training, and the game might have launched with a longer lobby timeout.

 

Further updates:

Today I qualified (i.e. signed up) 80 people between 1pm-1:30, for a 3:00 game.
I released a 40 person game HIT at 2:58 and immediately emailed the 80 qualifiers with the link.

Unfortunately, my game HIT had geo-restrictions of USA, while the qual HIT had North America, and I forgot to exclude people who had already played the game from the qual hit. In the end, only about 65 of the qualifiers actually saw the game HIT. (oops..)

Here's today's chart:


Within 4 minutes of launch (2 minutes of advertised time) there were 20 people in training or ready. Then, recruitment slowed noticeably. (I assume some of that is due to the HIT slipping down the page, but don't really have other visibility on why the uptake rate slows down so much...)

The game launched about 10 minutes after the HIT was released, at which point I paused the HIT so no new players would join and be disappointed.


What I'll do differently next time:
1. Set the exclusions properly on the Qual HIT.
2. Release the HIT right at the advertised time. The first players entered the platform less than a minute after the HIT was released, so I don't see a need to release early. Releasing at the advertised time might increase the rate of entry onto the platform slightly, as more people are ready to go, but this would mostly be just a simpler way to do it.
3. Increase the number of signups by about 1.5x, to be 4-5x the number needed for the game. I'd like to speed up the time in the lobby. As people entered the platform relatively quickly in this game, the only way to do that is to shorten the average training time. Without changing training itself, I could do that by recruiting more players. That way we won't have to wait for the slow ones to finish. (Note: this may bias the player sample, so I'll have to work out if that's ok.)

In the future, I'll be running 10+ games in parallel, so I'll need to sign up a larger number of players. The games launch as they fill, so the average lobby time will be much shorter, as fast trainers have plenty of other fast trainers to launch a game with, and slow trainers will arrive in the lobby at about the same time as other slow trainers. I'll alternate the treatment and control games, so that on average I have the same number of games with fast trainers as with slow trainers in each condition. Then I'll regress on training time within both the treatment and control groups, to see if there is an effect on my outcome measures. If not, no worries.

 

Thanks for this detailed procedure. I will use this in my data collection.

I do have a small question on the qual HIT: I believe it is expected that all the workers that have submitted in the qual HIT would be paid? In other words, is it really frowned upon if the requester only pays those submitted HIT only if the worker shows up in the actual HIT? I am just afraind that if the workers know that they can submit the qual HIT without the need to show up, some would not even if they said they would commit.

 

Yeah you should approve/pay anyone submitting the qual HIT. There's no way to know why they don't end up showing to the real HIT, and the qual shouldn't really pay enough to incentivize people to just randomly try to game it for money, so most people not completing the actual "main" HIT likely are missing for decent/good reason

 

Yup! I missed out on a $50, 1 hour Zoom interview because the proton therapy machine at the Proton Center broke down and I was stuck in the waiting room for 3 hours while they beat on it to get it going again. I had even rescheduled my appointment for earlier so I would have a small window in case something went wrong. Which it did, as usual.

I'd do a simple $.05 basic qual HIT. Age, gender, location, 'Are you actually going to show up?'. Plus, when you reject, it reflects on your approval rating and the lower that gets, the less likely you are to get quality data. If anyone even accepts your HITs at all

If your dataset is large enough, I'd suggest maybe having multiple sessions on a fixed schedule, so people who might normally would have done it they hadn't missed it by 30 seconds because someone was doing 64 in a 65mph zone for 10 miles. If someone misses the 6pm session they can catch the 8pm one.

 

I found the $.05 qualification HITs tend to significantly lower the reported average hourly wage for my requester account. I pay $6.5 on average for a 25-minute session (it is an interactive experiment so the payment would depend on the decisions of the workers and their partners), amounting to an hourly wage of $18. However, some workers report my qualification HITs with an hourly wage of $1 or $2, and because there are often more qualification HITs than the actual HITs, these reported statistics really hurt my reputation. I provide generous pay to build a good reputation and attract good-quality workers, so it is quite frustrating to see this.

Also, a large part of my HITs is bonus payment that is dependent on the worker's decision, which is about $5.5 (out of the total $6.5 payment). But some workers don't bother to report their bonus pay when they write a review, giving others an impression of low pay.

Is there any way around this? What can I do to make sure the workers leave reviews that better reflect the actual pay of my HITs?