Suggestion For Improving Mturk Suite

Discussion in 'mTurk Scripts & Resources' started by nmelssx, Jan 18, 2022.

  1. nmelssx

    nmelssx Turker

    Messages:
    36
    Gender:
    Male
    Ratings:
    +19
    It would be nice if Mturk Suite includes the requester's Hit Approval Rate over the past 2 years when you hover over their username when doing their hits. Hovering over the TurkView button next to the requester's name is very inaccurate, because sometimes TurkView will show that the requester have no rejections, but when you actually click on their name, it says they rejected almost 50% of their hits.

    For example, this requester:

    https://worker.mturk.com/requesters/A2WH8U9QRS2BBS/projects

    Has only a 46% approval rate over the past 2 years, but if you hover over TurkView's button while doing their hits, and it says that this requester has no rejections.
     
  2. Fuzzy_Dunlop

    Fuzzy_Dunlop Turker

    Messages:
    156
    Gender:
    Male
    Ratings:
    +58
    Two things to consider
    1. Requester does not have any project requirements and is likely to attract new accounts
    2. New accounts are most two categories of users
    2.a. Unfamiliar users submitting willy nilly
    2.b. Fraudulent accounts/nefarious workers

    With 2 reviews, my guess is that they are learning the hard way about the data quality problem on MTurk.
    Just an example of how that data point can sometimes not be as important as it seems.


    As for the MTS suggestion, it might get tricky. The approval rate on
    this page references the value

    Code:
    results[i].requesterInfo.taskApprovalRate
    and is only present in the JSON results from a search query from the main page.


    Results from main page:
    Source:
    Code:
    hxxps://worker.mturk.com/projects.json?filters[search_term]=UMN-IDEA
    Code:
    {
      "results": [
        {
          "hit_set_id": "3YVRJW0BXCUKS7ADY4YQ6MJAXC24ZQ",
          "requester_id": "A2WH8U9QRS2BBS",
          "requester_name": "UMN-IDEA",
          "title": "Manipulate a 3D...",
          "description": "Manipulate a 3D...",
          "assignment_duration_in_seconds": 900,
          "creation_time": "2022-01-22T05:05:51.000Z",
          "assignable_hits_count": 685,
          "latest_expiration_time": "2022-01-29T05:05:51.000Z",
          "caller_meets_requirements": true,
          "caller_meets_preview_requirements": true,
          "last_updated_time": "2022-01-22T05:05:51.000Z",
          "monetary_reward": {
            "currency_code": "USD",
            "amount_in_dollars": 0.3
          },
          "accept_project_task_url": "/projects/3YVRJW0BXCUKS7ADY4YQ6MJAXC24ZQ/tasks/accept_random.json?ref=w_pl_prvw",
          "requester_url": "/requesters/A2WH8U9QRS2BBS/projects.json?ref=w_pl_prvw",
          "project_tasks_url": "/projects/3YVRJW0BXCUKS7ADY4YQ6MJAXC24ZQ/tasks.json?ref=w_pl_prvw",
          "project_requirements": [],
          "requesterInfo": {
            "activityLevel": "Moderate",
            "taskApprovalRate": "52%",
            "taskReviewTime": "1 day"
          }
        }
      ],
      "page_number": 1,
      "num_results": 1,
      "total_num_results": 2
    }
    
    Results from Requester Page:
    Source:
    Code:
    hxxps://worker.mturk.com/requesters/A2WH8U9QRS2BBS/projects.json?filters[search_term]=UMN-IDEA
    Code:
    {
      "results": [
        {
          "hit_set_id": "3YVRJW0BXCUKS7ADY4YQ6MJAXC24ZQ",
          "requester_id": "A2WH8U9QRS2BBS",
          "requester_name": "UMN-IDEA",
          "title": "Manipulate a 3D...",
          "description": "Manipulate a 3D...",
          "assignment_duration_in_seconds": 900,
          "creation_time": "2022-01-22T05:05:51.000Z",
          "assignable_hits_count": 680,
          "latest_expiration_time": "2022-01-29T05:05:51.000Z",
          "caller_meets_requirements": true,
          "caller_meets_preview_requirements": true,
          "last_updated_time": "2022-01-22T05:05:51.000Z",
          "monetary_reward": {
            "currency_code": "USD",
            "amount_in_dollars": 0.3
          },
          "accept_project_task_url": "/projects/3YVRJW0BXCUKS7ADY4YQ6MJAXC24ZQ/tasks/accept_random.json",
          "requester_url": "/requesters/A2WH8U9QRS2BBS/projects.json",
          "project_tasks_url": "/projects/3YVRJW0BXCUKS7ADY4YQ6MJAXC24ZQ/tasks.json",
          "project_requirements": []
        }
      ],
      "page_number": 1,
      "num_results": 1,
      "total_num_results": 2
    }