How to create a small focus group about mTurk

Hi all,

I am trying to figure out the best way to create a small focus group of Turkers to briefly talk/chat ABOUT mTurk. This is the first of a series HIT's, each building on the previous. The purpose of the focus group is to generate a list of 10 or so tasks which would be considered inappropriate by Turkers (for example, completing offline tasks) or for structures and practices within mTurk that would be considered inappropriate (some hypothetical Amazon policy, for example).

This list will be used for a second HIT (rating the inappropriateness of the different tasks, structures, or practices; 50-100 Turkers). The results of the second HIT will then inform the third HIT (evaluating, in detail, on of those structures, practices, or tasks; 200-300 Turkers).

I don't know how to go about creating a HIT that would allow me to conduct a contemporaneous focus group, or if that would be even necessary. If y'all have any ideas, please share them. I essentially am trying to come up with an idea within mTurk which most Turkers would consider inappropriate (like, for example, a potential Requester making a thread like this?).

Oh, and I am an academic researcher. I research legitimacy within institutional theory.
https://en.wikipedia.org/wiki/New_institutionalism

Best,
A

 

If I'm understanding you want the same set of workers on each hit?
Give workers who do the first hit a qualification so only those workers can do the next hit

 

Hey there,

I've tagged your account with Requester flair, for anyone curious he is tied to an academic institution so this is legit.

You need everyone in the same place for this or would one-by-one be acceptable? Voice, video, or just text chat / survey based?

FWIW, this will probably never make the list haha. Most folks prefer it when Requesters reach out, especially pre-task, saves a lot of people headaches

 

Not quite. The focus group will probably be 5-6 people. It's just that what I learn from the focus group will inform the second HIT, and what I learn from that will inform the third and final HIT.

Gracias! I was wondering if there were ever people from Amazon pretending to be Turkers/Requesters scoping y'all out ^_^

I don't know if it is a matter of needing everyone at one place, just a situation that allows people to respond to each other since this fosters idea generation. I really have no idea what is considered appropriate/inappropriate within the norms of mTurk (both within mTurk as a whole or within the Turker subcommunity, if there is any difference). If I had time (and did more qualitative research), this would be something I'd love to explore substantively.

One thought:

Start a private conversation on this or another platform, ask for volunteers to participate. Post a HIT with a code, and give the code to the people in the private conversation, that way it doesn't have to be contemporaneous but there can be interaction.

 

@ChrisTurk this sounds like what Stanford does on slack can it be done here. It would be less complicated and probably be easier to direct workers to the conversation if it was here

 

You missed our conversation the other day about if asking for a zip code was identifiable and therefore a violation of mTurks TOS. Some of us were against it, some of us didn't care, and some of us gave false data to protect ourselves. Is this the kind of conversations you're looking to have?

 

This happens more than I'd prefer. In fact I'm well known for disliking a particular Requester who thought it would be cute to use the forums as his personal rat lab without permission or compensation for those of us thrown to the proverbial wolves from his experiment ^_^

(mostly kidding about the above, its just a running gag at this point )

You're certainly welcome and encouraged to leverage this platform however you can if it means folks get to participate in a HIT. We have one Requester using it similarly atm to recruit people for her work:

https://turkerhub.com/threads/schedule-a-hit-with-me-less-than-10-mins-4.947/

There are a variety of ways I can help facilitate things if it'll help and it turns out using TH would be useful for ya, just depends on the variety you need in your demographics (folks here are mostly American so if you're looking to query the sizable Indian workforce on mTurk I'm not sure how useful we'll be haha) and such.

One thing completely unrelated I'd definitely consider is maybe setting up a bonus structure for ideas that make the top 10. Or really anything that encourages people to contribute / brainstorm, otherwise you know how it is in group projects: one or two folks end up doing all the work LOL!

 

Save it for the focus group so that you can be compensated

But yes, in general, things like that.

Actually, in terms of getting things published, it is better to restrict samples to American; it's the easiest way to reassure reviewers that the participants are native speakers (even if there are plenty of non-native speakers in the US). It doesn't matter too much for the focus group since that's just about idea generation, but it would probably be important for the second HIT (selecting the idea) since that idea would be used in the third HIT (the application) and I'd want the population I am sampling from to match.

I like the idea of rewarding those ideas that get selected for Hit #2. Another possibility is using this thread as a focus group and some how rewarding 3.00 or something for every idea that gets included in the 10 that get selected? If that doesn't work, I'm inclined just to ask those who have participated in this thread, and then maybe if they can recommend 2-3 other people who are vested in the community and the idea of mTurk, to participate in a private conversation on TH. I don't think the total amount of time required of y'all would be too great, but it'll be spread out over a couple of days as people post ideas and respond to ideas in the conversation. I can probably justify 5.00 for participation and 1.00 for every idea included in the 10 (I'm the principle investigator, but someone else holds the purse strings).

 

sounds interesting let me know if you decide to do this

 

I would second the slack idea. You could setup a group with slack, which is a messaging/text platform, then recruit however many people you need and invite them to slack for the initial conversation.

People could use google docs to generate their ideas and then link you to their google docs.

When the work is done with the initial 5 or 6, you could setup a HIT for compensation that only they can accept.

For the larger group I would use a qualification HIT so you don't have to setup individual HITs for everyone.
You can do a qualification HIT for a penny, then probably just bonus everyone when the task is completed.

 

Hah, you'll find no arguments from me that American is the way to go on mTurk, just wanted to make sure I was being as honest as I could be about the demographics of workers here

As far as American variety we have a good cross-section of folks you'll normally find on mTurk (stay at home parents, IT folks, full-timers, etc), but they're also far, far above average in terms of experience & familiarity with the platform. Kind of the nature of taking it seriously is learning about it, and that's generally what we promote here. Again, I'm assuming that's good for your first part, but if you're looking for nativity for some of the psych manipulations this is a bad place to come lol.

Either way would probably work, just depends on how organized you're looking for. God bless ya if you opted to but sitting and watching the daily thread on any given afternoon would probably suss out the 10 most common complaints pretty quickly haha .

If you want a shortlist of folks who I know would put in a good bit of effort & give a decent cross-section of experience with the platform I can work one up and send it your way for you to contact them. Even without it though you'll find more folks who are willing & able to contribute meaningfully to something like this than you could ever possibly have a budget for haha.