1. API Information

    The TurkerView API page has instructions as well as contact information for support in getting set up with our new API infrastructure.

    You can also email us directly at hello@turkerview.com if you run into any problems!

    Dismiss Notice

GeoHIVEs Data Quality / Qualification / Wages Discussion

Discussion in 'Requesters' started by Bobby, Jun 8, 2018.

  1. WildFlower

    WildFlower Survey Slinger

    Messages:
    4,080
    Gender:
    Female
    Ratings:
    +4,653
    I'm not one who speeds through, but that's also why I didn't work much on the recent 0.10 batches. I'm happy I likely provide more accurate data, but 10 images for 0.10 checking diligently for 3 values per image and not speeding leaves me with a poor hourly. Adjusting the image count down to 5 or upping the pay is a necessity.

    Also, I'm not a big fan of a new qual test for workers who have already done hundreds or thousands of HITs for you. I'd be up for existing workers being granted the new qual and then implement the tiered system and see how it goes. If you aren't getting the workflow you need on the batches at that point, then it seems like a good idea to open up a qual test for new workers.

    Also, I've had quals that adjust before and one of the main things that needs to happen if you go this route is that it needs to adjust both ways, not just deduct points. If you get something wrong it needs to deduct a point per whatever number of HITs you determine, and if you get something right it needs to add a point per whatever number. You can also just have someone needs to monitor cases that are falling near the disqualification line and reset the number if the overall work product is up to the standard you set. The latter has worked fine for me in the past as long as there's someone viewing and responding to messages when they are sent after workers notice they are getting too close to the disqualification number. Whatever approach you choose, I think being transparent about it is best.
     
    • Like Like x 6
    • Love Love x 3
  2. Randomacts

    Randomacts Survey Slinger

    Messages:
    77,425
    Gender:
    Male
    Ratings:
    +106,148
    I still have that in my copy of chrome......
     
  3. angel

    angel Survey Slinger

    Messages:
    9,917
    Gender:
    Female
    Ratings:
    +15,754
    I don't have time to read the whole thread right now but just skimming through here are my first thoughts.

    personally how fast I can do one of your hits depends on what I am being asked to look for. I have not done Geohives in awhile (just seem to keep missng them) but I remeber awhile ago there were bridge ones that I was able to fly through because for most of them it was easy to spot if there was a bridge or not. That said I did slow down when I got images from urban areas where I had to look more carefully.

    assigning a value to the qual that drops and rises according to quality is a good idea say a value of 100 is the starting point and 75 is the minumum needed to do work a warning that a worker's value has dropped to a 90, 80 or whatever will be a heads up to them that they need to fix something.

    I always assumed you had "golden questions" built into the batches if not these are a better indication of quality then speed
     
    • Like Like x 1
  4. Celtic Lady

    Celtic Lady Survey Slinger

    Messages:
    2,868
    Gender:
    Female
    Ratings:
    +5,223
    I also like the idea of a qual value that goes down as you miss the items on golden maps. If there is some way to raise it after getting X number of golden map items correct as well that would be great. It's kind of frustrating to only see a qual like that go down with no way to raise it back up so feedback on what I got wrong would be very helpful.

    I love working on geohive and pfa hits. They are actually some of my favorites that I've done on mturk. PFA is my highest paying requester by total $s and GH is 6th highest. I'd like to think I'm pretty good at them but in all honesty I don't really know if I am because I haven't gotten feedback about it. A variable qual score would at least let me monitor the value and see how I'm doing.

    I also don't think that speed alone should be a reason to reject. Again, golden maps can come in to play here and spot checking work of people suspected of scamming in one way or another. Depending on how fast the maps load I can be working on 2-6 hits at a time and sometimes I can be submitting all 6 of them in rapid succession just because of timing. I try and only work on the batches that I know I can do well. Sometimes I test one or two of a new batch and say, "naw, not my thing this time" and stop if it isn't worth my time or I don't like it well enough to be confident I can give good data.

    TL;DR speed alone is not something that I think warrants a reject or qual revoke. I can spot a swimming pool or solar panel on a map like nobodies business these days. Variable qual score and some feedback on what we're doing wrong is the way to go, IMO.

    Thank you for taking the time @Bobby even though you aren't on the GH team anymore, really appreciate it.
     
  5. jdzane

    jdzane Survey Slinger

    Messages:
    9,198
    Gender:
    Female
    Ratings:
    +8,625
    Golden questions like pinterest would be good, if you must go the changing value qual like ProductRnR please make sure you a) explain what makes it go up and down and b) let us know when that happens.

    Someone with more time/knowledge can explain my refrences.

    Oh and they took away our hot keys (1-pool, 2-no pool, 3-too cloudy, etc). For those of us who can't write scripts, those were a godsend, and now that they're gone, that's why I left too.
     
    • Like Like x 1
  6. ToBeDetermined

    ToBeDetermined Turker

    Messages:
    38
    Gender:
    Male
    Ratings:
    +32
    I don't post many messages here, but I figure I would put my random thoughts out there. I have only done PicsFromAbove tasks so far, but I do want to eventually get into GeoHive. PicsFromAbove is kinda random on whether it is worth trying or not, and I want to see a similar task be more reliable than that, if that makes sense.

    I think its a good idea to put in golden questions into the task. It would help control bad data and keep workers on their toes. Pinterest tasks actually informs workers that they have made mistakes at the end of the task, and if it happens enough, the worker is temporarily locked out that specific batch. If you can implement that system fully, it would stop alot of bad data before it's submitted. Also the server would see a decrease in overall load as workers get locked out of batches. Another small thought, I am willing to do extra images in addition to what the tasks usually have. Like if its 15 images for 15 cents. I would do say 18 images for 15 cents, 3 of them being golden. I don't want a system like that to mess with the profit margins too much.

    The moving qualification value is also a good idea, but I'm not sure if it would be 100 percent necessary if the golden questions are successful on their own. An issue I see with the moving qualification is that someone would have to balance it so the qualification isn't too easy or hard to lose. During the initial rollout of that qualification, community feedback and hot fixes will be very important.

    What I would like to see is a qualification test that drops quarterly. Once people have the qualification, they can get temporary locked out of batches if they miss golden questions. Get locked out of enough batches, they lose that quarter's qualification. Lose enough of the qualifications, then get locked out completely. Seems kind of lenient, but each time someone gets locked out, there will be an opportunity for communication, which can be used to improve guidelines and quarterly tests.

    Anyway, those are my random thoughts on this. I know that any of this would be hard to implement. I don't know how much control GeoHive has over the task, But I am glad that they reached out to the community.
     
    • Love Love x 2
    • Like Like x 1
  7. PDR

    PDR New Turker

    Messages:
    19
    Ratings:
    +13
    They should be able to sandbox it here https://requester.mturk.com/developer/sandbox and you should be able to work on it here https://workersandbox.mturk.com/ without it costing them anything.
     
    • Useful / Informative Useful / Informative x 1
  8. laby

    laby Survey Slinger

    Messages:
    9,739
    Gender:
    Female
    Ratings:
    +26,745
    This should be a given but it would also be nice if whoever is in charge of dealing with hits now would make an account here and keep an eye when there is a batch up. You're gonna get better work if there is someone around who can clarify things if people have questions. @Bobby shouldn't have to be called back from wherever to deal with stuff that's no longer his job.
     
    • Love Love x 3
  9. oldtimeturker

    oldtimeturker Well-Known Turker

    Messages:
    891
    Gender:
    Male
    Ratings:
    +1,772
    I haven't done a geohive in years, because i don't think they're worth the pay by any means. But a system with golden questions ala pins, would be nice and just locking people out of batch if they miss one. Unlike pins tho it'd be nice if you showed the one that people missed on. I swear with half the pins I dont actually miss one yet get golden showered.

    edit: always appreciate requester communication. thanks for that
     
    Last edited: Jun 10, 2018
  10. bubbleoh7

    bubbleoh7 Guest

    Ratings:
    +0
    Maybe focus on tuning your HITs with the goal of discovering quality workers who will then produce quality data.
     
    • Like Like x 1
    • Love Love x 1
  11. James Wilson

    James Wilson GeoHIVE

    Messages:
    27
    Gender:
    Male
    Ratings:
    +98
    Hello folks. My name is James, and I'm trying to fill Bobby's shoes on the GeoHIVE team. I've been reading through your comments, and many of you are disappointed in the user interface. Please bear with us as we continue to grow and improve it. We welcome feedback and suggestions. What did you love about the Tomnod interface (key bindings, zoom functionality), and what did you hate (slow loading images)? People are also disappointed in image quality for this campaign. Again, we are testing new things. Some of you may be concerned that we will revoke quals or reject payments if you make a little mistake, especially on one of the hazy or dark images. There's no need to worry. We understand it's difficult to tell the difference between rail and road and other feature types, and it's even more difficult to discern those in a poor quality image. We are adding a tag type called "Poor Image". This tag type is there to give you confidence in us. If you feel like it's impossible for you to mark any features because of image quality, drop a "Poor Image" point on the image to let us know.
     
    • Like Like x 7
    • Love Love x 2
    • Useful / Informative Useful / Informative x 1
  12. Eisenpower

    Eisenpower Survey Slinger

    Messages:
    4,840
    Gender:
    Male
    Ratings:
    +11,843
    @ChrisTurk
     
    • Like Like x 1
  13. Eisenpower

    Eisenpower Survey Slinger

    Messages:
    4,840
    Gender:
    Male
    Ratings:
    +11,843
    This is just my list, but...

    Pros:
    Key binds
    Examples of the query in the instructions (without forcing you to see it on each HIT)
    Ability to go back to previous image
    Unsure/Poor Quality/Etc. option

    Cons:
    Slow Loads
    Multiple Tags, especially without reflecting the pay
    No communication on rejections, qual revokes, etc.

    Some people like the zoom function, but it never really helped me for anything. Zooming in on a fixed quality/size image just made things blurrier for me and just wasted my time. If that isn't the case, then maybe it would be helpful. Glad to see y'all are trying to communicate again though. That's always important.
     
  14. Manderwiz

    Manderwiz Active Turker

    Messages:
    105
    Gender:
    Female
    Ratings:
    +211
    Hi, James! Thank you for the new tag - it's one thing that has kept me from even trying these. The other being too much work for too little pay. Looking for one feature in ten images would be acceptable, but having to look for multiple features is just too much even for the person (like me) who enjoys working with satellite images and wants to turn in good work. Also zoom.
     
  15. Moa711

    Moa711 Active Turker

    Messages:
    419
    Gender:
    Female
    Ratings:
    +253
    Hey James, I had to create an account just to chime in here. I have been doing this hits, and hits like it, for years. Back then they were Nates I think. I have a question concerning some of the tags, more specifically the main road thing. Some images are a ton of roads. Sometimes there is no real main road eg, looks like a gated community with no real main roads, and then some seem to be a whole slew of "main roads". Are you wanting us to tag all the " main roads"? And what about the images that seem to have no main road, and just a bunch of back roads?

    I am glad to see the leniency on the road/rail thing. There are times it is just impossible to differentiate between the two.It can be difficult to see bridges at times too.

    I have to agree with what others have voiced that the pay could be better, or the amount of tags could be less, like break it down into just roads, or roads and bridges, but not everything that is there now. And if you are wanting every road we can find being tagged, then it really just needs to be roads in one hit.
     
    • Like Like x 2
  16. James Wilson

    James Wilson GeoHIVE

    Messages:
    27
    Gender:
    Male
    Ratings:
    +98
    That's a great question. Just place 2 or 3 Major Road points per image, one on each major road you see. Roads and rails are really lines, not points. We're not asking you to follow the road, just place a single point on it. Also, we're increasing the circle radius so that you can only place 2 or 3 road tags on a single image in the future. And yes, major road. That's a bit subjective because a two lane paved or well defined dirt road in the desert is a major road whereas an interstate is a major road in an urban environment.
     
    • Like Like x 3
  17. James Wilson

    James Wilson GeoHIVE

    Messages:
    27
    Gender:
    Male
    Ratings:
    +98
    As for pay, I understand there's 5 things to look for, but in many instances only one or two of those features are present.
     
    • Like Like x 2
  18. Eisenpower

    Eisenpower Survey Slinger

    Messages:
    4,840
    Gender:
    Male
    Ratings:
    +11,843
    There is still a cost to being aware and searching for those things. I actually stay away from HITs that ask you to look for a lot. Even if you end up not finding anything for most of them, you still spend energy having to go through and look for those. A lot of requesters don't understand that. You're probably thinking about it in terms of the data you receive while we're thinking about the work involved in providing that data. That's why it's also a good idea to actually try out the HITs as if you were a worker. It's a good way to bridge the gap in understanding the other side. And if you think about it like an assembly line, it's much easier/faster to have one worker focusing on one thing, and progressing it down the chain. Having extra responsibilities on one requires more time because of the extra work. Extra time/work is a nope when I have a choice. That also plays into the part about people trying to maximize their earning potential and speeding up in order to make it worth it. There's a consequence to each of those strategies, and we all decide what's more important to us. That old mantra that someone mentioned of "You Can Have It Good, Fast or Cheap – Pick Two" rings true a lot for me.
     
    • Like Like x 7
    • Love Love x 2
  19. turker

    turker Survey Slinger

    Messages:
    4,440
    Ratings:
    +6,199
    could someone please tell me what the AA is on the current 23 cent geohives, the older batch of them. thanks!
     
  20. ezdzd

    ezdzd Active Turker

    Messages:
    519
    Gender:
    Male
    Ratings:
    +979
    First and foremost, you seem like a pretty decent guy and despite the fact that you're in Tampa(Brandon FL immigrant), what I'm about to say is in no way personal. I have an analytical mind and often I can offend people and this is not what I want to accomplish here. It is offensive to insinuate that people are doing hits to fast. The platform is built around workers going at their own pace. I know that you suggest recording a video but I'll touch on that later. I heard a saying along time ago that has always stuck with me "People will do what they are paid to do". A janitor paid 7 dollars an hour performs differently than a janitor paid 20 dollars an hour. If the companies business model cannot sustain the lower quality of work I suggest bringing in warm bodies at minimum wage and paying out Federal taxes and social security . The lack of good data from work that is too fast is a result of the labor market at its finest, a true showing of capitalism. If you pay people substandard wages you will get substandard results. There is no debate here. If you offer wages at a certain level the labor market will rise to that level and not above. The most egregious statement was suggesting that workers were gaming the system. Pay taxes on every worker you employ at the federal minimum wage, first and then we can talk productivity. To suggest that workers aren't giving any thought to the results they provide is like telling 10 year old they aren't swinging their pick axes fast enough in the mines in 1818. If your company cannot survive with the substandard wages that are given to mturk workers, that is not the workers fault. Plain and simple you need to charge your own clients more for the work you are providing. If you don't know how to do so or feel like your team lacks the negotiating prowess, I'd be glad to help for a fee. The labor pool you are using as your supply is responding to the capital you are offering it, and the results cannot be surprising. To suggest that workers provide a screen cast of their work is absolutely fine, as long as you provide them compensation for doing so, and I have a feeling you won't. The workers are absolutely skirting the line between good work and getting as much money as possible, Rockefeller would like a word and so would his dependable janitor who makes 20 dollars and hour. There cannot be a comprise nor should there be in the the market for labor. It will rise to the level of productivity relative to the pay offered. Once again I'm not ranting or blaming or even upset. I don't even do your hits(they don't pay enough). People will do what they are paid to do. These workers are not employees, they are at will contract workers, who often respond to hive mind mentalities against requesters. I'm not suggesting radical change, but I am suggesting that you realize the labor pool from which you are depending your business on. No hard feelings I promise.
    -D
    turker since '13
     
    • Love Love x 5
    • Like Like x 3