At the moment I am testing some HITs on MTURK. I have been surprised that they were not scarfed up as my testing showed I could do them in 20 seconds or so each and I set the initial payment at $0.20 expecting that it would take a non-expert in the field to take 4 or 5 times the amount of time it takes me. The name of the HITs is TEST COLLECT 10K MDA 2. I know one mistake I made was to require a Master's qualification. But I would be curious about any other thoughts folks have as to why these were not completed quickly. This is a test run so I can sort out the time issue to set some reasonable expectations about cost and quality. When I was watching yesterday I noticed that the average time spent was less than a minute. It is a small batch as I do not want to get overwhelmed by noise as I try to figure out the parameters. For instance, I can't afford to pay that rate and have two workers complete each task so if I am going to pay that rate or near it I have to sort out how good the work will be and how to filer out the mistakes.
Well if they have a master qualification most of us can't see them. They haven't given out master since December 2015 and it was given out randomly at that. Throw some of them up without a master qual and we can give you some feedback. Oh and what is your requester ID so that we can find the hit.
Part of the reason that you're not getting a fast response is probably because of the batch size. Most of us search for hits available based on either dollar amount or number of hits and if either of those are small it gets buried. Amazon quals are a set of hurdles that are likely better to avoid since you can create and assign your own quals as you see fit. My second suggestion is the name of the hit. It doesn't describe what the work is so a lot of people will probably skip over it. Coming here is a great start though. Alerting people what the batch work is and when it's posted is much more effective to make people aware of your work. We all routinely work batches in that dollar amount but it's because we're watching for specific requesters so getting to know people here, explaining what you need, and answering questions will go a long way.
I can't comment specifically, but the Masters qualification immediately eliminates a large section of the worker pool. Another thing is if you happen to be a new requester without any TO ratings, some people will only do a few hits at first to see if you are fair or not because 1) you don't get paid for rejected work and 2) rejections can hurt our percentages and affect our ability to qualify for other work.
Thanks for all of these comments. They were helpful, I will make some tweaks tomorrow based on these suggestions. That issue about Masters is really interesting because Amazon does tout Masters on the Requestor interface as a reasonable way to make sure you have folks who are paying attention. Interestingly I was watching it closely on Saturday and approved the initial ones that were completed right after they were completed so as to encourage whomever had worked on them. I am going to have a decent amount of work over time. My big concern is sorting out how to manage the work well as it comes back in. That was the big reason I initially posted only 29 hits (58 with having each hit done by two different 'Turkers'. I think tomorrow or Wednesday I should be able to carve out enough time to post and check 1,000. When I do I will mention it here again and then see what happens.
Welcome to TurkerHub! Thanks for dropping by to get feedback, I've tagged your account here with flair to help distinguish you as a requester in case you pop into the daily thread to post about your work (they can get kinda hectic haha). The typical "starter kit" quals that most new requesters are recommended to use (by workers and requesters I've spoken to in my time on the platform) are: Approval % > 98 (be careful with this, mTurk is somewhat inaccurate and will not recognize anything beyond whole numbers, so if you set it to > 99% hardly anyone will be able to do your work since even someone with 99.999% approval couldn't accept it) HITs Approved > 5,000 Location is one of US, CA, UK, NZ, AU (these are the predominantly English native language countries on the platform). You can restrict it to just the US if you want, kind of up to you on if that's necessary or not (if you're doing work that familiarity with US systems like addresses/phone #s or something it may make sense to do for example). The combination above is usually sufficient to give you a sizable worker pool (that still rules out the worst performers) that can be somewhat counted on to produce results that are good enough so you can further tune data from. For some work it'll be good enough altogether. Hard to give specifics beyond that without seeing the project/HITs themselves, so I'll keep an eye out for them tomorrow and make sure its shared to the daily thread as well There are a lot of different approaches to this that range from closed qualifications to "golden answers" to consensus building (mTurk has built in support for this), etc etc - the best avenue largely depends on your HIT and desired involvement level with it. A lot of mTurk is "get what you put into it" from the requester end, but its a very good platform once you've got the hang of it This is a great video to see approaches and learning processes used by an Amazon requester. Its long but worth the watch to get a feel for how they approach the platform. I don't agree with all of it, and I've met requesters who use parts of their methodology and have come up with their own as well, but its definitely a great intro to the platform from a requester's perspective especially in regards to data quality.
Thanks for all of this. I will use this information and try again. I want to note that I have a small company and while this is a priority I may not be able to get things set up again until tomorrow or even possibly Thursday. Another question - should I extend this thread when I have additional comments, results or questions or should I start a new thread? Thanks
For general discussion about your work you should probably just extend this one. When you are actually ready for people to try your work or you've made a big update here you'd like people to see I would recommend saying something in the daily thread. Today's can be found here... https://turkerhub.com/threads/05-23-totally-tuesday.448/
I hope this is acceptable. I am posting below a link to the HIT template I will be using on the platform in a couple of hours. I thought maybe I should make sure there is no confusion and try to answer any questions that might arise. I did an initial test earlier in the week, I benefited from looking at the results as well as the comments posted above. I will note that the average time to complete the first batch was a bit under 60 seconds. When I am in the grove I can do more than 2 of these a minute - but I do have to come up for air periodically and I have been been doing this for some time. Model HIT The correct results for the model hit above BEGIN: 767 END: 1147 My initial post will be a little bit under 2,000 HITs. I have about 20,000 total for this round. I need to do it incrementally though to make sure I can have these reviewed and processed quickly so there is no lag on approval. Please provide comments about the task clarity etc.
Can you post your requester name or ID so I can add you to my watch list? I appreciate the example HIT, makes it much easier to figure out ahead of time, these seem super straightforward.
My requester name is Burch Kealey. I will make sure to indicate when I actually post them but I am going to wait and watch for the FeedBack
The example looks pretty straight forward. Is it the same piece of information you will be looking for in each hit?
I took a look at your hit template and it seems just fine to me. Other than that I really won't be able to say until you put up the test batch and i have a chance to really work through a few.
looks pretty straight forward, will defiantly check out when you post a batch. The sample ahead of time was defiantly helpful thanks.
At a high level - yes I am looking for the same piece of information in each HIT - the starting and ending points for the ITEM 7/6 disclosure and when that is not present the starting and ending points for the reference to where the disclosure can be found.
Sorry I am having some permissions issue with the files I put on S3 to be displayed in the iFrame. This is not going to happen today.
No worries, take your time the smoother the launch the better for your data anyway If you need any help feel free to let me know, I have a modicum of experience with the requester dashboard, CLI, and API. Nothing super extensive but I've fiddled a bit with all of it to get some perspective from y'alls side.
Thanks Chris - I am good. My mistake was to try to use the console and not the API. Everything is loading on AWS right now it it seems to be right. I will post again once everything is set.
