Turkerview with mobile only HITs: requester mass rejects, yet workers aren't reviewing on TV

Discussion in 'Help & Guides' started by SolidWork, Dec 2, 2019 at 5:09 PM.

  1. SolidWork

    SolidWork Turker

    Messages:
    45
    Ratings:
    +11
    A recent batch with a requester, in this case Lars Vinberg, with a 'mobile phone only' HIT, served up mass rejections, after having a good history as a requester.

    TurkOpticon has 7 new worker reviews, just yesterday, about this requester, all HIT REJECTED reds.

    Turkerview, has no updates.

    Turkerview needs to be installed and running during the work, and doesn't offer an alternate method of reviewing manually. I've heard others say they install scripts on their phones, although many of us keep that on our PCs for work.

    Is there a way for workers to manually leave reviews on Turkerview? If not, I request this feature, if the script writer is reading. Thank you!
     
    Last edited: Dec 2, 2019 at 6:47 PM
  2. Tripsa

    Tripsa ฏ๎๎๎ฏ๎๎๎ฏ๎๎๎ฏ๎๎๎

    Messages:
    9,952
    Gender:
    Male
    Ratings:
    +22,810
    Just checked that profile and there is one report of a user having their hit rejected. Could be that those that are on here and got rejected by the same requester have yet to leave a report about it.

    They're a couple of users on here that got rejected and were waiting to see if they got a reply from the requester inquiring about the rejection before they left a review in hopes that the rejection would be reversed.
     
  3. PurpleWolf

    PurpleWolf Active Turker

    Messages:
    457
    Gender:
    Female
    Ratings:
    +342
    He has responded to some people and has said he's taking a look at the rejections tonight, and will be reversing the ones that were rejected incorrectly. He used a semi-scripted process to review them, and some were automatically rejected due to an error.

    Edit: I'm not sure if your post meant you had a problem with the requester or not, so my comment may or may not be helpful. I would assume the people who primarily use TurkerView (instead of that other one) simply haven't reported their rejections yet to give the requester a chance to correct the problem. Lars has been on Mturk quite a while, and in my experience, has been a fair requester. It's unfortunate when workers are quick to trash an established requester's reputation over an error rather than give them the chance to make it right.
     
    Last edited: Dec 2, 2019 at 6:15 PM
  4. SolidWork

    SolidWork Turker

    Messages:
    45
    Ratings:
    +11
    I appreciate your responses.

    I'm specifically saying that a review on Turkerview cannot be left, because many of us do not load Turkerview on our cell phones, and may not want to load scripts on our only phone line. Lars Vinberg is an example, yet my reason for posting is the mechanical issue.

    I'm unsure there is a mechanical method to leave a Turkerview review, if it's a cell phone, and there's not Turkerview installed. I'm asking if that's possible manually, because I want to.

    I used the Lars Vinberg example because the obvious disparity between reports on Turkerview and TurkOpticon was very clear (Turkerview no updates, TurkOpticon - 7 red HIT REJECTED reviews in one day), and I want to make sure Turkerview is able to mechanically get accurate info. Even on cell phone HITs, without Turkerview installed.

    If there's a way to leave a Turkerview review for a cell phone HIT w/o Turkerview installed on the cell phone, I'm interested! Thank you.
     
    • Like Like x 1
    Last edited: Dec 3, 2019 at 12:11 AM
  5. PurpleWolf

    PurpleWolf Active Turker

    Messages:
    457
    Gender:
    Female
    Ratings:
    +342
    Ahhh, I see what you're asking. I did misunderstand your original post. That's a good question. I don't "think" it's possible, but I could be wrong. Hopefully someone else will come along and give you a more definite answer.
     
    • Like Like x 1
    • Useful / Informative Useful / Informative x 1
  6. ChrisTurk

    ChrisTurk Administrator

    Messages:
    44,469
    Ratings:
    +113,272
    It's not possible, the review would be missing far too much crucial information w/o the script. I wouldn't really recommend ever installing TVJS on a cell phone and submitting reviews through that though. The times would be so negatively skewed versus normal desktop users it would likely end up throwing an account into an outlier status? Idk if this is in regards to mobile turking or just for this HIT but worth mentioning I think it'd end up having negative implications.

    I don't think purely mobile restricted HITs are really worth the consideration, and in this case even it was covered by someone (God bless 'em), but I'm really not super knowledgeable about mobile turking tbh so maybe there is a % of people strictly on mobile worth covering in some way but a full blown mobile app is probably out of my knowledge sphere lol
     
    • Like Like x 1
    • Useful / Informative Useful / Informative x 1
  7. Yuk

    Yuk Well-Known Turker

    Messages:
    2,225
    Ratings:
    +4,202
    For anyone that cares, you can install scripts on phones with a tampermonkey app and run them through Dolphin and it behaves the exact same way as if you were on desktop.
    One could submit reviews through TVJS on mobile this way

    Dolphin
    Tampermonkey for Dolphin
     
    • Useful / Informative Useful / Informative x 2
  8. SolidWork

    SolidWork Turker

    Messages:
    45
    Ratings:
    +11
    I appreciate the feedback. Sounds like unless you install the scripts on your phone (which I'm not choosing to do), Turkerview is not going to get that portion of information. I still use a few sites to check out requester reviews, like this example.

    It would be a good option to offer in Turkerview, for situations like this, to go directly to Turkerview and manually enter a review. Turkerview does a lot already that I'm grateful for.
     
    • Like Like x 1
  9. Ornac

    Ornac Survey Slinger

    Messages:
    6,363
    Gender:
    Male
    Ratings:
    +5,978
    @ChrisTurk could probably speak more to this, but I think a lot of the reason to force having the app installed is to ensure that the user did the hit and that the information in the review is valid. Although you lose out on some information you are protected from a lot of false or exaggerated information as well, it seems a good compromise to me. I think Turkopticon definitely has its place, especially for newer workers who need to be much more careful about rejections.
     
    • Like Like x 1
  10. Masterbah

    Masterbah Well-Known Turker

    Messages:
    2,879
    Gender:
    Male
    Ratings:
    +2,966
    I mean if the requester only wants you to use your phone, then it's fine to have a skewed hourly because it represents what most people will go through with the task.

    FYI, you can install tampermonkey, TVJS, MTS, etc on firefox on mobile phones and run them all just like on your computer.
     
  11. ChrisTurk

    ChrisTurk Administrator

    Messages:
    44,469
    Ratings:
    +113,272
    When TV first started you could enter manual reviews. It is objectively an awful method of reviewing for a platform like MTurk.

    Outside of the outright fake data that gets submitted users are generally not capable of submitting highly reliable data without it being pre-filled/tracked for them and TO offers no data validation because they have no clue what the worker interface even looks like lol. Screwed up requester ids/names/hits/yadda happens at a 10x higher rate from waaaay back when TV had a manual review form and it is just awful to try to clean up (+ frustrating both for reviewers and users).

    But the fake data is by far the worst part. Folks just make up stuff. Even w/ TVJS people still do it, but I generally know when they are so it's easier to learn/identify/track/improve on.

    No one has any idea if someone on TO is even a worker or requester. They've had issues with research teams going in and impersonating workers to run studies on the platform and just shrugged it off and let workers eat the headache so how does anyone know if X/Y/Z is even from a decent source to begin with? All of that stems from the lack of control & data validation because it opens the door to anyone with a half baked idea & a dream.

    TV is absolutely without question not impenetrable to manipulation (it happens, I see it weekly even from folks who think they're clever about it) but the barrier is infinitely higher and the only real cost is this 1/300,000 type of task. Even at that cost, TV's profile for Lars is still more fleshed out, and statistically mirror's AMT's own data, than his TO profile.


    PlatformReviewsRejection %
    AMT-4%
    TV721.3%
    TO24 (3 w/ absolutely no info?)38%
    That's just not representative of working with that requester. Yea, I get a rejection stings & such, but his TO profile is useless if you have >200 HITs because 9/10 when you do his very well paid HITs, you'll get paid for it.

    I do this constantly, and I cannot see a reason to use TO once you can eat any single rejection during the day and be fine. Even then.. AMT's statistical data is more helpful. Want to just refuse to work for any requester that rejects anything? AMT's data is better than TO's (and TV's).

    Can they change pattern? Yeah sure. But it's incredibly slim chances esp for requesters w/ good profiles.

    Just my opinion/thoughts on the "manual review" process and why I've never once considered adding it back in since removing it. Not that I'm not open to the discussion (it's been had multiple times over the years & I'm always down) but there needs to be really compelling reasoning for it IMO.
     
    • Useful / Informative Useful / Informative x 2
  12. Ornac

    Ornac Survey Slinger

    Messages:
    6,363
    Gender:
    Male
    Ratings:
    +5,978
    I kind of forgot they added this when I made my post. It does kind of make TO redundant when you consider the Amazon data and TV.
     
    Last edited: Dec 3, 2019 at 12:39 AM
  13. ChrisTurk

    ChrisTurk Administrator

    Messages:
    44,469
    Ratings:
    +113,272
    Blah blah CT is biased & a jerk but I legitimately don't think any of the traditional use cases for TO stand up anymore:
    • "TO has more data"
      • Nope.. It's not even close anymore. Let's ignore that a solid portion of TO's "data" is just SPAM (no ratings, that type content that can't be submitted on TV to begin with), and that a portion of it is 5+years or older, TO1 has 469,000 reviews all time (give or take) and TV is at 550,000 + the tens of thousands of return reviews if you just really want the TO type content.
    • "TO has more historical information"
      • I mean.. maybe? If you think reviews from 10 years ago are relevant than alrighty, but the majority of requesters coming back are in under a 1 year cycle so TV covers this nowadays
    • "Rejections bro"
      • Just look at AMT if you want hard data, and TV's profiles tend to more closely mirror AMT's sets for actual nuance/discussion
    I don't care if someone continues to load TO and does their thing, but it's a waste of time on a platform where 10-15 seconds can swing the hourly wage on a HIT an entire tier :dunno:

    TO2 doesn't count, their API has been offline for almost a year so I have no clue why anyone is still throwing things into the black hole that is that place :emoji_joy: workers can't even retrieve it afaik
     
    • Like Like x 1
    • Love Love x 1
  14. Ornac

    Ornac Survey Slinger

    Messages:
    6,363
    Gender:
    Male
    Ratings:
    +5,978
    I'm pretty sure you know that wasn't what I was saying. :emoji_sweat_smile: I was saying I was probably a bit biased because I hang around here all the time. :emoji_joy: I agree, like I said I didn't really think about the new Amazon stats when I said TO had it's place for newbies.
     
    • LOL LOL x 1
  15. ChrisTurk

    ChrisTurk Administrator

    Messages:
    44,469
    Ratings:
    +113,272
    Nah I know but what happens is I make a post like that and someone will go "OMG CT IS BASHING TO" and I'm not trying to bash TO any more than I'm trying to bash researchers who found out 3 years ago worker ids were connected to AMT profiles.

    It's just outdated & no longer relevant information (IMO) so I like to try and throw my POV out there but inherently I type poorly and it comes off as rudeskis so I throw in the disclaimer lmao.
     
    • Like Like x 1
  16. Ornac

    Ornac Survey Slinger

    Messages:
    6,363
    Gender:
    Male
    Ratings:
    +5,978
    Gotcha. I read that a little wrong(in my defense it's having a couple of PBRs and playing video games time). Was like OMG I wasn't trying to offend you(I probably should have assumed Randomacts would have done that a long time ago if someone was going to do it):emoji_joy:. I can't imagine what it's like having your comments scrutinized like you do. Better you than me, but it's great that you step up and do what you do anyway. :emoji_thumbsup::arms::cheer:
     
    • LOL LOL x 1
    • Love Love x 1
  17. ChrisTurk

    ChrisTurk Administrator

    Messages:
    44,469
    Ratings:
    +113,272
    Super easy.. just throw out a "I'm a moron don't listen to me" disclaimer on everything then close the tab anytime that doesn't work.

    Works 2 out of 3 times, guaranteed.

     
    • LOL LOL x 2
  18. lars

    lars New Turker

    Messages:
    1
    Ratings:
    +5
    This is lars, the requester.
    we are having issues with
    1) people doing more than one hits
    2) people copy and pasting their workerid wrong.

    We are replying and reverting the rejections at uber human speed, so please be patient.
     
    • Love Love x 4
    • Like Like x 1
  19. ChrisTurk

    ChrisTurk Administrator

    Messages:
    44,469
    Ratings:
    +113,272
    FWIW MTurk allows you to pass the worker's id yourself so you can solve both of those problems w/o relying on workers.. which kind of ties into our off track discussion above w/r/t relying on users entering data haha. If you need help working with it for future HITs feel free to shoot me a DM or email (hello@turkerview.com) sometime & I'm happy to throw docs/code your way. Sorry never done the HITs myself to give more specific advise but it is something simple that can remove a world of headache for all parties :)
     
    • Like Like x 2
  20. awdrf

    awdrf Well-Known Turker

    Messages:
    4,457
    Ratings:
    +4,785
    I spoke with you via Reddit DMs just a little while ago, about being the ex-TO mod. Glad you found your way over here. I can vouch ChrisTurk is a great resource.

    As I told you in the DM, I wouldn't worry too much about your TO score, while our fearless administrator is undoubtedly biased, he is absolutely correct about it too.
     
    • Love Love x 1