Hello, After launching a survey on MTurk that links out to Qualtric software (with workers entering a completion code back into MTurk for payment), I noticed an issue with lots of clearly duplicate responses of workers taking the survey multiple times and submitting for payment. Since I have it set on MTurk to only allow unique workers to respond, they must have multiple MTurk accounts from which they are taking the survey and getting paid. This is seriously affecting the validity of my data. Anybody else encounter this before and have a solution? Thanks!
Qualtrics offers IP detection as a default. Do you have duplicates? You can also detect VPN's. Not a requester, but that seems like a good way to filter out spammers. https://www.qualtrics.com/support/survey-platform/survey-module/survey-checker/fraud-detection/ Also, what qualifications do you have set on Mturk?
Thank you for your response. Regarding qualifications, I have HIT >97% approval rating and >5000 submitted. Any others you would recommend? I have the Qualtrics thing on but they can get around it by simply clearing cookies...
That doesn’t sound very secure on Qualtrics’ part. You might want to consider trying Prolific or even CloudResearch (though I know some people aren’t big fans of theirs). MTurk can definitely feel like a bit of a wild west at times. If you can narrow your audience, here are some settings that might help tighten things up: Total approved HITs >= 10k HIT approval rate (%) >= 99 Location In: US From what I’ve seen, most reputable Turkers have a 99% approval rate and well over 10k approved HITs. A rough rule of thumb is that it takes about 1,000 approved HITs to offset a rejection, so spammers might not meet these stricter criteria. Adding something like a VPN detector or recpatcha's throughout could also help https://www.qualtrics.com/support/s.../survey-checker/fraud-detection/#BotDetection. Another good option is to use a closed qualification. You could start with a shorter, lower-paid screener survey designed to detect fraud (e.g., include some writing tasks). Then, assign the closed qualification to the few workers who pass and let only them access the main survey. This approach might take a little more time upfront but could save you a lot of trouble with data quality.
