1. TurkerHub is now TurkerView Forum!

    To better integrate the services & streamline user experience we've moved the forum to TurkerView's domain. From now on you'll only have to log in once across both services and you'll have access to all the features available on what used to be two different websites.

    If you notice any bugs please let us know!

    Dismiss Notice

Help on defining Rewards and timings

Discussion in 'Requesters' started by Roland Gude, Oct 30, 2018.

  1. Roland Gude

    Roland Gude New Turker

    Messages:
    3
    Gender:
    Male
    Ratings:
    +1
    Hello Everybody,
    I am currently investigating how we could use Mechanical Turk at our company. A key concern for us is, that we do not want to exploit our workforce and thus want to set adequate rewards for the HITs we post. Obviously we are restricted by a budget though. Currently we have no real experience with Mechanical Turk and i wanted to hear the thoughts of people actually making their money with it. I suppose you have a whole lot of experience and some things you expect from a requester.
    First of all I'd like to give you a little bit of context for what we are trying to do.
    Given some PDF documents which contain (amongst other things) tabular data we want to extract from that tabular data certain fields. Unfortunately the documents come in all kinds of layouts etc. And the fields are not canonically named. So we might be looking for a Number called Thingys per Square mile and in some document you would find a row labeled "T/mileĀ²" in another it might be "T per mile-x-mile" (you get the picture). And it is perfectly possible that the table does not contain the correct numbers at all. Usually each PDF is about 3 pages long, has some pictures, some text and the interesting table. The table contains between 5 and 15 entries of which we might need 7 (always the same 7).

    I suppose this could be broken down into different HITs in several ways
    i.e. one HIT "find the table"
    one HIT "extract the numbers just from the table"
    or alternatively one HIT for getting it all done in one Go.

    Here are some of our assumptions (pleas correct us if we are off here)
    Tagging a table in a document should be doable in roughly a minute
    Extracting the numbers from the table might take up to 5 minutes (due to awkward naming, or missing numbers)
    Neither task strikes us as very difficult
    Forums suggest that one should pay at least 0,10$ per minute estimated time required (if our guesses with time required are right, we might be able to pay 0,12$ per minute)

    Do these assumptions seem right to you guys? Are we missing something important?
    Please share comments on it, as we would really like to set up the project in a way that is acceptable for all parties involved.

    Kind regards,
    Roland
     
  2. angel

    angel Survey Slinger

    Messages:
    9,646
    Gender:
    Female
    Ratings:
    +15,403
    It is hard to say with out actually seeing a sample task but a few things that come to mind

    1.assume it will take a worker unfamiliar with what you are looking for longer than you estimate
    2. how a hit is laid out and clear instructions will greatly impact completion time and over all worker experience
    3. And it is perfectly possible that the table does not contain the correct numbers at all.
    what is the worker expected to do in this case, if the information is not available to extract will they still be paid fr their time or are they expected to return the hit and not be compensated
     
    • Like Like x 1
  3. Tracy Wethington

    Tracy Wethington Survey Slinger

    Messages:
    4,179
    Gender:
    Female
    Ratings:
    +5,115
    Thanks for considering Mturk for your hits. Have you ever looked at Turkerview? It's a huge database of reviews from turkers that can give you insight to what is good/fair/poor pay and reviews on the hits themselves. That will give you a great idea of a starting point for fair wages. Good luck, thanks again for considering mturk.
     
  4. ChrisTurk

    ChrisTurk Administrator

    Messages:
    31,631
    Ratings:
    +79,739
    Yes - an age old adage about assumptions :p hahaha

    In all seriousness, you seem to be approaching this in mostly the correct way (especially revolving around breaking up the tasks, it will increase the overall cost a bit but it will also decrease your error rates).

    The nice thing about MTurk is that you can quickly iterate & test your assumptions here at very little cost. Just throw small test batches and measure how long it takes workers to complete them. Or as a safe rule of thumb if you're familiar with the task and it takes you 10s, assume a worker will take 2-3x that until they also become familiar with the task.

    You're also welcome (I highly encourage it) to put the HITs up on the sandbox & ask workers to review them for you, it'll save you the headache of having negative reviews about your profile from the get-go on turkopticon/turkerview while also giving folks something concrete to give feedback on (it is much more difficult for us to draw a vision of your work based on your description than it is for you :p )

    Oh man, gotta be a little blunt here for clarity sake I'm sorry:

    No forum existing in the last few years should be suggesting this. That recommendation is over a decade old and from a time when minimum wage was like $5/hr. Certainly nothing on this forum will suggest this - 10c/min is entirely unacceptable as a pay rate here (where American workers come to work) - if you can get the work done by outsourced labor I have no qualms with your business but I just want to be clear here we do not promote those pay rates for any reason (mostly should someone doing research on google stumble here).

    Given that workers assume your tax burdens for the work I do not recommend paying anything less than $15/hr for batch work if you require American workers (for English proficiency, for legal reasons, whatever the case may be). Again, if you can get it done within your preferred error limits more cheaply elsewhere then it is what it is - just want to clear up confusion on that old payrate suggestions as its a ghost that never seems to die haha.
     
    • Like Like x 5
    • Love Love x 2
  5. Epictetus

    Epictetus Active Turker

    Messages:
    612
    Gender:
    Male
    Ratings:
    +692
    You also may want to look into crafting a good "qual" (qualification/training hit or batch) oriented toward ensuring that you have a ready pool of workers that you KNOW has the knowledge and capacity to do your work, the way you want them to do it.

    Conscientious turkers are always on the lookout to earn as many quals as possible, so as to be able to find good work whenever we go online. A prestigious/lucrative qual is something precious to have and we are loathe to risk any of them by submitting sub-standard work, so you will benefit as well.
     
    • Like Like x 4
  6. angel

    angel Survey Slinger

    Messages:
    9,646
    Gender:
    Female
    Ratings:
    +15,403
    This
     
  7. urtho9

    urtho9 Active Turker

    Messages:
    950
    Gender:
    Male
    Ratings:
    +707
    As someone who trains people regularly at work and works directly with all different sorts of people on a daily basis in a social services setting, I have a very important suggestion. Give several examples of what the work should look like and have it in as close to the format of how the HIT is setup as possible. Both images as well as youtube examples are great and quite simple to put together. The best way to look at it from an efficiency perspective is that any random Joe off the street should be able to look at your training materials, having no clue what all this is about, and still based on the examples given be able to complete the work with little to no issue. One of the biggest shortcomings of mankind is communication as we tend to interpret things differently. If you provide some seriously concrete visible examples of how to do the work, this can alleviate a lot of this difficulty barring some of the potentially less common and more difficult examples. Be sure that you are straight forward in what your expectations are.

    :emoji_triangular_flag_on_post:By communicating effectively in the beginning this can also help your long term budget because you won't have to continually respond to stupid questions that some random guy is emailing you about, when you could be doing something more important and valuable with your time. This will make the platform even more effective than it might otherwise be. From the initial contact you sound like someone with integrity who wants to do this the right way. Hope this helps.
     
  8. Roland Gude

    Roland Gude New Turker

    Messages:
    3
    Gender:
    Male
    Ratings:
    +1
    Hi again,
    Thanks for the feedback, I honestly appreciate it.

    This is a very good point. My gut feeling says that if the number is not there, not finding it is the correct answer and thus would be paid. I suppose that not finding a number is actually more time consuming then finding it though so it might still be annoying to the workers a little. But they should indicate that the requested value was not in the table and get paid.

    ---

    Thanks for suggesting it, i will check it out.

    ----

    Ah, good to know. We will probably try to do that.

    Thanks for being blunt. I was asking because I was not able to obtain this information. A clear answer is exactly what i needed.
    Regarding the pay, i was wondering about that already as 0.10$ per minute seemed low. From a business case perspective this is of course the kind of numbers that make the case attractive, but it does not feel right to us. So we definitely need to have a go at the numbers again in order to get pay rates that are acceptable.

    could you provide pointers on how to craft "good" qual batches? and how this works? i did not encounter any information about this yet.

    Yeah it helps. You are right providing good visuals should be possible.


    Thanks again for your valuable feedback. We will now have another round of planning internally considering all the input we could get and see where we land.

    There is another thing i wanted to ask.
    The volume of this is initially not very high, There might only be three or four documents to process a day at the beginning. We sure hope though, that this volume will steadily increase but we do not know for sure when that will be. Would this be an issue for workers?

    Kind regards,
    Roland
     
    • Like Like x 1
  9. urtho9

    urtho9 Active Turker

    Messages:
    950
    Gender:
    Male
    Ratings:
    +707
    Not really easy to give you a good answer on this one, it depends very much on the worker and situation.
     
  10. angel

    angel Survey Slinger

    Messages:
    9,646
    Gender:
    Female
    Ratings:
    +15,403
    This is a very good point. My gut feeling says that if the number is not there, not finding it is the correct answer and thus would be paid. I suppose that not finding a number is actually more time consuming then finding it though so it might still be annoying to the workers a little. But they should indicate that the requested value was not in the table and get paid.

    while it may be correct that not finding an answer takes a bit longer you would be surprised how many times as workers we are expected to just return such a hit and absorb the time/cost ourselves so the option to mark the hit as data not available or whatever and submit will go a long way to improving workers perceptions of you. I for one factor in this "unpaid" work when calculating hourly and this can quickly make it not worth my time so glad to here you feel workers should be paid for their time.

    Unfortunately I'm sure there are some out there who would take advantage of this and use it to "Scam" the hits but this is another area were setting up a qualification hit helps as serious workers are careful not to jeopardize their qualifications plus you are less like to get random fly by the night workers who are out to make a quick buck and not thinking of the work in the long term.

    Usually for qualifications a small batch is put out with known answers for workers to do and those who do well are granted a qualification to work on other batches. From your perspective this also makes it easier to control data quality since if a worker drops below whatever threshold you set you simply remove their qualification,This is preferable to workers as well since the other option is to block them through Amazon and that negatively impacts them across the platform and it is likely just a matter of them not being good a particular type of hit.

    As for the amount of work I think being upfront and communicating will help with that. Perhaps an announcement here or through email when you are posting one of these small batches so those with the qualification know to look for it, this would be especially important if the batches are not posting regularly.
     
  11. rater9

    rater9 Survey Slinger

    Messages:
    1,959
    Gender:
    Female
    Ratings:
    +3,155
    One thing that would be very beneficial for those workers who are NOT on the Eastern Time Zone is that qualification hits and/or regular batches are posted at such a time that is is not the "middle of the night" so to speak. 8:00 AM on the East Coast is 5:00 AM on the West Coast and by the time a West Coast worker trips out of bed and to the computer a good portion of batches and especially qualification hits are gone. There are many times when I begin my day at 3:00 AM in order to try to catch work while it is still available.
     
    • Useful / Informative Useful / Informative x 1
  12. Roland Gude

    Roland Gude New Turker

    Messages:
    3
    Gender:
    Male
    Ratings:
    +1
    Sorry for keeping this thread stale for a little while, unfortunately other stuff came up.
    I will look into the qualifications. It seems quite important.
    regarding time zones, we probably have little influence on that, as we need to submit the batches when they are ready. I suppose that will be throughout the day in CET. I am not sure though and will keep that in mind. Thanks for the pointer