Apologies if these are dumb questions, but I'm having a heck of a time finding direct answers. I have 3 HITs currently available that are set to expire tomorrow (1/18/19); I didn't think to change the default availability when I set them up last week (newbie mistake). HIT ID: 3OPLMF3EU6IBRK8I4FP8MXM0VWVNLD HIT ID: 3ZQX1VYFTE0YUYOBB5GHWH0Z161O87 HIT ID: 3P0I4CQYVZ29KZKT98WNAX3LJD5OWN I prepaid for 400 assignments per HIT, but I have not reached this goal (I'm at 279, 193, and 333, respectively). I would like to extend the batches so the survey will be available longer for more workers to complete; however, when I try to do this, Amazon tells me I do not have sufficient funds to do so. I do not understand why this is--I am not attempting to purchase new HITs, I am attempting to extend the time available for workers to take the HITs that I already purchased. What happens to the unclaimed HITs (and funds used to pay for them) once the batches expire? Do I get refunded for the unclaimed HITs and then can re-post the batches for the numbers i still need? If so, how do I avoid having Workers who took it the first time take it again? Many thanks in advance here~
Are you doing this through the requester dashboard, CLI/API, or another managed service? Yes, anything not sent to workers when the batch expires will be taken out of "hold" (you can see this on your req account overview: https://requester.mturk.com/account ) and freed up to republish the work. Download a list of results, take all the worker IDs, create a custom qualification & grant it to all of those worker IDs. Before republishing the batch, set the project to DISALLOW workers with those qualifications. That'll prevent them from accepting the HITs (provided you set it up correctly)
Mturk help is sort of unhelpful: https://www.mturk.com/help "How do I receive a refund for prepaid HITs? If you would like a refund of your prepaid HIT purchase, please contact customer support." I can't see any of your hits now, are they supposed to be invisible without a special qual? The way most people prevent re-taking is to issue an exclude qual to the test subjects so they can't accept any hit with that qual. EDIT: Never mind CT actually knows what he's talking about, please ignore me.
FWIW, this doesn't apply to her situation. I know it's a little confusing, but this is referencing the ability to get the money deposited into your requester account back out into the original bank account. Unlike worker accounts where you can withdraw to AP & then to your bank, once a requester deposits funds there is (or used to be) absolutely no way to get the money back if you didn't need to spend it on HITs for w/e reason. It used to be a hard "no" from Amazon or they'd do it but close your account, now they do it on a case by case basis but only with manual support approval (which is what that is referencing). She doesn't want/need to withdraw the money back to her bank, just apply it to a different set of HITs. Which wont be a problem at all since the money is only put on "hold" while she has HITs up & the hold is lifted once the HITs are cancelled (or withdrawn to Amazon's bank account if they are submitted AND approved). Ex: I've got 3 penny HITs up, and $5 in my account. If I wanted to recoup the 3 cents, I'd just need to cancel the 3 hits & the funds would be available again to apply to another prepaid hit set. No need to fuss w/ MTurk support. If I wanted $5 back into my actual bank account, I'd need to contact support.
@Epictetus -- thanks for noting that MTurk help is not very helpful! Makes me feel slightly less like an idiot. @ChrisTurk -- many thanks for your guidance; this is incredibly helpful. My HITs expired this morning, and the unclaimed $$ is back in my account, so I'll be re-publishing the 3 HITs that didn't fill @Epictetus -- yes, the HITs used some premium qualifications (gender and political affiliation). I did this to try and secure equal #s for each gender * political affiliation group. I filled female liberals no problem, and am at about 85% on male liberals, but am having a heck of a time filling the conservative surveys. I know not all Workers fill out those qualifications, so I was certainly missing folks likely eligible. For both of you (and anyone else following), I'd love your thoughts here: when I re-post the HITs, should I continue using premium qualifications, or is there a better strategy for getting conservative-identifying respondents? I used the premium quals to (a) avoid the sample being overrepresented with liberal respondents and (b) avoid wasting the time of Workers not fitting my target demographics. My hands are a bit tied with respect to making major changes to the survey, as I really don't want to have to go back through IRB again (but I would if it was totally unavoidable). Again, many thanks -- this forum has been so helpful!
I'll try to break down your three main options here: Amazon Premium Quals A lot of workers don't particularly like those quals because they're expensive for what they give you, however considering the demographic you're trying to reach (which is very much underrepresented on MTurk vs general populations) it might be a good idea to just stick w/ it so long as you aren't horribly constrained by time. Reason I say that is because despite the price workers can't really "game" that particular qualification AFAIK. Once they report a political leaning, MTurk doesn't let them change it (and while it's possible for folks to flip/flop or change their minds, for all intents & purposes it's a more "permanent" status than your other option). Self-selected demographic pools What you can do if you are pressed for time in completing the study is set up a "demographics survey" in which you ask a few different questions. Don't simply ask political leaning because you'll get people who lie to try & qualify (that's just how it is online, sadly). Honestly if you can work around asking that particular question at all it might be best (instead ask who they voted for or something). To be perfectly frank I'm fully ignorant here on if this would be kosher for the purposes of the study, but you get the point: don't divulge what you're looking for because if you do people will gladly give it to you even if it isn't true. Truthfully, I recommend doing it from a 2nd requester account & just using the results to grant qualifications on your main account if you do go this route since your name already is tied to what you're looking for, especially clever workers might put 2+2 together. I'd say it's unlikely, but minor effort for a clean dataset. The (best) middle ground TurkPrime offers a lot of these demographic groups already filtered/ready to go. I'm not going to say they're 100% accurate, but they will likely be "cleaner" worker pools than you might get attempting to do this yourself for the first time with no experience/familiarity with the platform. They do track workers who flip-flop their answers a lot & take surveys they shouldn't & disqualify them over time so you'll filter out a lot of the riff raff & stolen accounts. As a bonus, it's pennies on the dollar cheaper than using MTurk's premium qualifications. TP charges something like 5% base fee and I think half of that for academics. As far as value goes TP is by far the best "bang for your buck" you can get for online survey platforms. It's a nice mix of the gigantic userbase of MTurk w/o the downsides & it's miles better than supporting garbage like Prolific. I'm not affiliated with any of them, but realistically they (TP) do provide a lot of value to our ecosystem for what is a pretty fair price so I'd recommend them as a convenience service for sure.
@ChrisTurk -- this is awesome, thank you SO much for your insight. I'm not too terribly pressed for time to finish this, beyond just the general nerdiness of wanting to wrap up the surveys and start digging into the data. I think I have to have some kind of descriptive report to my funder by April. I don't know anything about TurkPrime, but I'm now certainly intrigued. I will be reading up on this asap. Thank you again!
