HIT Forker Update

Discussion in 'News & Announcements' started by ChrisTurk, Jan 22, 2019.

  1. ChrisTurk

    ChrisTurk Administrator

    • HIT Forker has been updated to function with TurkerView's new View API [details here]
      • If you already connected your API key w/ TVJS, you don't need to take any further action. Forker will read it automatically.
    • Everyone will need an API Key starting Feb 1 in order to use TV with HIT Forker.
    • The first month is free for every user. Once that first month ends the price is discounted at $5/month for users that sign up before Mar 1 and $7.99/month otherwise.
      • NOTE: If you have submitted at least 10 reviews on TV (as of 1/20/19), you will get a permanent key for free, this covers hundreds of users who helped build TV & is also extended to a # of community members who have contributed their time/expertise elsewhere.
    • Forker now has more reliable access to TV's dataset, meaning about a 40% increase in coverage across the board from less failed requests/etc.

    Improvements (Why this will make you more money)

    This might sound like a "big deal, who cares" post but I want to highlight the fact that it is actually a huge improvement to both reliability & coverage. While Forker has always "had access" to TV the caching of data & single instance server meant that requests failed regularly. On top of that Forker waits for a response (or failure) from TV/TO before updating it's interface which means users awaiting responses from API servers (TV, TO, etc) do not get updated results in real time (or nearer to it). These kinds of things are "hidden costs" of working on the platform, users can comiserate with seeing a HIT appear on their scraper, clicking on it & instantly seeing "No more HITs available" - faster responses from API requests will help improve overall response times of scripts like Forker.

    Unless you know about these kind of issues it's difficult to understand how (or if) this is costing a user money but on TV's end these issues will be cleared up inside Forker.

    To see best results, I recommend only enabling TV's API in Forker as it will still wait for responses from other API endpoints & delay processing of the UI until they respond (or fail). Generally TV should have excellent coverage:


    Overall, though, if you use TV's dataset inside of HIT Forker you'll now be seeing the whole thing in real time. No more clicking on the website & seeing different ratings than what is displayed in the scraper, and the scraper should have a little more vibrant feel since TV's data tends to be weighted more heavily towards top tier work & you'll have access to more of that data now.

    View API Load Testing / Optimization:

    TV's API should be pretty elastic, however there are many aspects of the servers/infrastructure I cannot optimize until I see just how it responds to Forker+TVJS being on the same endpoint.

    This small window of time before Feb 1st is probably my last chance to test this kind of stuff, and I'll be measuring the API with a few monitoring services, but I'd also like user feedback. If you see this in TVJS:


    Please let me know by simply sending me a message w/ the time! It shouldn't happen often (and if it does, it shouldn't take me more than an hour to throw up more servers to handle it) but I did go w/ a conservative resource estimate to start out with so I can get a clearer idea of what the usage:resource ratio is really like before problems occur.
    • Love Love x 3
    • Today I Learned Today I Learned x 2
    Last edited: Jan 22, 2019
  2. germainrene

    germainrene New Turker

    Interesting ideas, I like it!