funny and annoying

Fellow requesters,

Sure, there are bad seeds and cheap folks on the requester side. But not all workers are up and up too. When we pay fairly, it attracts not only great workers but also some cheaters too. Sadly, the effort they need to put into so that they won't get caught easily is not necessarily minimal. I'm puzzled by the fact that it is actually easier to just answer our darn survey truthfully like most good reviews here than casually cheat and think they may get away with it. Of course they can get away with it sometimes. Nothing is fool proof. But when we catch one, we always have enough evidence. Internet is no longer the wild wild west and it's so easy to use metadata to figure out a lot about a given data entry point. Sadly, the casual cheaters don't really realize that and would smear us however they can. Guess what, it is a gig labor market, you don't have to work for us and we don't have to hire you. Better spend the time to actually do honest work will boost your income faster.

Do you run into the grumpy workers in your experience? How do you deal with them? Any fun stories?

 

https://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/BlockingaWorker.html

Blocking offending workers with that reason given

and a quick message to amazon with something to the effect of:

"Hi, these worker IDs knowingly attempted to defraud my HITs by misrepresenting their demographics... I know this because ... evidence given"

Only Requesters can prevent forest fires bad actors on the platform.

 

A new phrase was given to me not too long ago...
"Context is King"
And with your commentary here about "bad workers", I refer to above. As (you) a requester, making a blanket statement about "bad workers" and such, eh....that isn't a super great thing to put out in the open. It somewhat will turn people against you.... quickly. I get you get scammers/spammers, I feel for you. But coming to a forum and basically blanket putting workers on point...not so cool. There are times I have been rejected for a hit from a "good" requester because I didn't spend the (literal) 3 seconds that another worker did, so *MY* work was rejected. How about those of us with the copy/paste weirdness where at times the copy/pasting of codes fails and you wind up pasting something from who knows when. Coming here and expecting some sympathy is a bit tone deaf in nature. Coming here and then stating "Guess what, it is a gig labor market, you don't have to work for us and we don't have to hire you" is just passive aggressive in nature and really uncool. Maybe this is just my view, but I would never work for you after this posting.

 

Sorry I offended you. As you said, context is king. But I don't think my post "blanket" put workers on point. The review forum makes it easy for workers to highlight bad requesters. But, honestly, there is a need for requesters to communicate about their blacklist too, even if 99.9% of the workers are legitimately good and hard working people.

What I did not say in my post is that, 99+% of the workers who helped us have been great without problem. We have to reject a few for every project, knowing fully that we will get some emails and we will re-approve most of them because I always believe the chance of false positive. So we want to communicate. clarify and re-approve if possible. Sadly, most workers only respond promptly to rejection notice. We can block all rejected workers and move on, but that is not how we operate. We spent lots of effort to respond to individual inquiries promptly and explain lots of details about how we do what we do. I treat this as real work so I communicate professionally regardless the manner of any inquiry coming my way. You may be communicating with requesters professionally. But some workers do not and you wouldn't know much about it because requesters could block them and move on.

Furthermore, if you ever work on our HIT, you would not be rejected by us with the examples of human errors you gave here. We may need to reject you to get your attention for the clarification we need so that we can re-approve you. Since I don't know your MTurk ID, feel free to try to experiment it by taking a screenshot of the verification code but then enter your MTurk ID to the HIT. I promise you that you can witness our SOP first hand. But you need to know that, mismatch is mismatch, it may be a simple error for you who complete the HIT honestly but it may be a cheater who accept and submit the HIT within seconds and just entered the MTurk ID hoping requesters may not catch it. There is no difference from our dashboard to say workers are honest or cheating. .

In addition, even most of the cheaters who know they got caught would not waste time trolling us or making noises to get unwanted attention. Obviously we would block those caught by us for future HITs. The funny and annoy thing is, every once a while, some cheaters just think they can't be caught, and keep complaining until someone hears and agrees with them. So I just want readers to know that, this is a two-way street. I heard from workers complaining how so many requesters are very cheap and did not pay workers enough. I make sure our HITs come with fair incentive even if you spend a longer than average to complete it. So, from my perspective, when I receive a grumpy complaint from an obvious cheater, I would not be too sympathetic because I'm not the one exploiting the contract of the HIT. If you cannot understand my perspective as a requester on a thread for requesters, then it is what it is. I'm not posting to get sympathy. I just want to hear if there are interesting stories from other requesters along these lines.

 

Spoiler: My definition of funny and annoying

 

What an interesting thread. No stories to share, sorry - probably because we don't reject/block at this stage. But, as A6_Foul_Out pointed out, blocking a worker can help to increase the strength of the platform.

However, the reason we don't block is for the same reason Azazael points out. Sure, sometimes it's obvious a worker was just trying to game the system... But not every single time. The same goes for bonusing: if a worker doesn't meet a given criteria for getting a bonus (we use more than one factor), is that because they were acting illegitimately, or because they just didn't meet that criteria for some reason?

I wondered I the issue was legitimate workers who just looked like they were gaming the system, but then saw your follow-up reply RE communicating with workers. I guess it's still possible there are some people who look like they're cheating but aren't, but it seems less likely now. Those few angry cheaters must be frustrating to respond to!

Sorry for my derailed response. Tl;dr: I don't have any funny stories to share.

 

People who are only workers only have a very limited perspective of the requester side. They don't know how bad it can be and some cannot comprehend the economic impact dishonest workers can have.
The us against them point of view from a minority of vocal workers and easily offended workers creates a hostile environment for discussion and progress. The emails from a very small minority of workers would be unbelievable to all dedicated and professional workers. They are dealing with a cross section of the population which includes lazy, entitled, threatening, cursing and outright insane emails. This causes requesters to not be responsive to messages and some leave the platform. Even calling some workers professional and dedicated elicits negativity from some workers who believe that working on this platform is a joke. This platform got me through the 2007 recession, fully paid for two new cars, a down payment for a home and has made me in excess of 100k for my time. It has been a godsend for me and my family.

When a requester allows participation from below the 99% approval rating on hits, the garbage floods in, at least 10-15% of unusable responses. If they go below 95% they can count on at least 20-30% of the responses being either scammers or people who have limited knowledge of the English language.
If they use the block feature that Amazon suggests they use to police their platform, they will get blasted on worker review sites and forums.

I have stories but will not post publicly but I will tell you how I deal with very small percentage of problematic workers. I approve everyone then qualification block them for future participation. I never respond to complaint emails and unless someone is trying to outright scam like brute force submission of 200 hits when only one is allowed, I never block. It is not my responsibility to police Mturk.

 

No worries. We mostly only reject people without a matched verification code and matched ID. We even accounted for one or two off digits/characters with the ID. We only asked for the redundant MTurkID entry to make it slightly difficult for bots. Whenever workers emailed us in advance saying they made an error, we don't reject them upon initial approval.

We do not discriminate in our response by the manner, tone or the degree of professionalism (or nastiness). Those who knew they wouldn't be able to provide the information to corroborate would not bother. Those who appeal after they got the rejection notification are mostly re-approved when they offer a quick note that corroborate with the data pattern we saw.

We allow a degree of 'grey area' in our process because we care more about compensating those who may be caught by being false positive and we are fine with compensating some false negative cases. This is why I'm not sympathetic to those correct positive. Someone tried completing three times (or 200 times) when we only allowed for once. Someone got caught and was not happy and then making lots of fuzz. This got my curious about whether other requesters may run into the same thing. It's easy to ignore them. But would you want to know about these IDs so that you can block them in advance to preserve data quality?

 

Thank you for letting me know that there are normal readers who can understand both sides of the phenomenon. We certainly block those bad seeds and did not re-approve the dead-on cases. Those IDs certainly will not be able to get enough work as their site analytics deteriorate.

 

I can't imagine someone trying to complete a HIT more than once.... argh. Frustrating for all involved, I'm sure.

Thankyou for the offer of the IDs. I'll leave it for now, but appreciate the offer. We are currently using different methods to preserve data quality, although I'm starting to wonder if blocking is on the cards.

I'm really intrigued by the idea of qualification blocking problematic workers. We do use qualification blocks to prevent workers from taking a study they've already taken (if re-uploaded), and to prevent workers from taking a study that's too similar to one they've done in the past, but I hadn't;t considered using a qualification to block workers before...

 

When you use the Block feature, occasionally it will trigger Mturk to send a nasty email to the worker/participant stating that you blocked them and that their account is in jeopardy if they receive too many blocks. This can impact your reputation as a requester and can impact the workers ability to work on the platform.
The additional csv file can be transferred to any requester accounts you upload it to even if the participant has never worked on that account, the qualification will still attach.

People make mistakes but I have found through trial and error that most who are put on my qual block csv will be repeat offenders if they are not added to it. So if the person submits bad data, I pay them for the hit/hits and add them to the qual block. They are paid once and their reputation is not damaged from a rejection but they are missing out on the future work/studies from any account or requester I work for.

 

from a worker prospective this is the preferred method. The requester gets to protect their data and even "punish" the worker with out jeopardizing their ability to work for other requesters. Unfortunately not all requesters know they can do this, some even use blocks for to prevent retakes of the same or similar surveys and other situations where they are absolutely inappropriate.

 

Worker harm and reputation harm were my biggest fears with respect to using the block feature, the former much more than the latter.

Looks like I'll be investigating making a block-qual soon. Thankyou all!!

Edited to add: Angel, I feel I did not thank you enough for sharing your perspective as a worker here. It's easy enough for me to sit at work and try to guess what workers might like, but it's infinitely more valuable to have the kind of feedback and information you've shared. Thankyou again!

The same goes for you, Billy and Coolpat.