Hello forum

Apologies if this post falls afoul of any rules. I didn't see anything against it in the guidelines but do let me know if there is a problem with it.

I am from an early stage AI startup (for credentials, we are part of EF) seeking to make online advertising better - despite funding many free services online, ads are mostly awful. We think the reason is that it's currently too hard to manually write & design interesting ads for every group of people they target - thousands of target segments for each ad campaign. We've built an AI system that can generate creative online advertising and as such, we hope will contribute to a better internet experience for everyone online.

Our system requires lots of labelled data as well as some information on who's doing the labelling, and in some interesting ways goes beyond what Mturk provides to requesters as an HIT platform. So we have built our own basic HIT platform and are looking for 2-5 workers to try out a beta. While this is not our core business, we are keen to support the worker community because we know how crucial you are to many new AI startups. If successful we'd be looking to make the platform available to other cutting edge AI startups in a way that meets the needs of early stage businesses and the workers who want to be part of their visions of the future.

All we're asking at the moment is for your thoughts in this thread on whether you think a new HIT platform for this purpose is a good idea, what might make it attractive for workers and whether you'd be keen to be part of our beta.

Thank you!

Will

 

I've seen some really amazing things done on mTurk, so I'm really curious what you mean by this? Requesters have successfully categorized the demographics in their worker base on mTurk without much fuss in the past, including targeted advertising testing across half a dozen companies I've seen come/go.

My personal opinions on the questions:
New platforms generally suck, and only really help out the person saving on the fees Amazon imposes. Reinventing the wheel & all that.

Compensating folks well for their time & the value generated for your company makes it attractive, so details on that would help folks a lot in regards to volunteering to sign up for something that currently doesn't exist.

FWIW, you're in a space where a new "mTurk killer" pops up every week, and the details are always next to non-existent, so don't take any of this the wrong way but hard facts & such are really helpful because most everything that comes through in this regard is just a half assed attempt to get VC funding, pump an ICO, etc etc then disappear :dunno:

 

Thanks Chris, really appreciate your thoughts.

In terms of hard facts do you mean things like - is the platform easier/faster to use, does it pay more, are there more interesting jobs, more visibility of longer term jobs than Mturk? If so I can share a link to the platform beta when it's ready and folks can feed back on something specific.

As mentioned this isn't our core business so we're not looking to be an Mturk killer, just get the worker metadata we need (which is deeper than demographics and goes into daily advertising profile and requires lots of redundant labels), yes save on $$$ that's not going to workers and also give workers something better than what's available on things like I mention above. We're not doing ICOs or anything like that!

If workers generally have a massive preference to use the major platforms like Mturk and we'd be wasting time to offer something else that perhaps solves some niche of issues for workers then that is good to know too.

Cheers

Will

 

I like the idea of an opportunity to try something new and would be willing to try your beta, but to be honest with you mturk is the staple of my daily workflow when it comes to task work. Unless something is significantly more attractive in terms of time for money trade-off, I am usually unlikely to head off-platform.

 

I am always open to another income stream as working outside of my home is not an option for me but some information on how steady the work will be and how long it is expected to last would be useful. Some things that pop up as mturk alternatives do not have enough available work to make them anything more than a distraction. For example one that has popped up in the last week or so never has any work available when I check so eventually you stop checking as it is only taking away from time that could be spent working to be worth while there has to be enough work for the people who are signed up. I don't mean it has to be a sole income source or anything like that just enough work to make it worth the time to be there for x amount of time during the day.

 

Obviously, the main thing I'm going to care about as a worker is the income potential. Secondary to that, the work being interesting helps if the tasks take very much time at all. I would definitely be open to trying out the platform and giving feedback on it.

 

I'd just be parroting others at this point, but like they've said

1) pay
2) platform (for workflow purposes)
3) It certainly doesn't hurt if it's interesting


I'm also curious, so when you're live, I'd check it out.

 

I mostly meant like "does it even exist yet?" haha. But details like:

• What is the target hourly wage workers will be paid for tasks?
  
• Will beta testers be paid for time spent or tasks completed only? Betas tend to be buggy and poorly put together from the worker perspective. Even if mTurk has major shortcomings from the worker perspective we've put in thousands of collective hours fixing/improving it for our own community. So, if folks are running into issues, providing feedback that is then used to increase the desirability of your platform, etc etc are they paid for this? How? If not, why not?
• Is it open to US workers? We're majority US based here, your profile says Singapore - if its open to US workers are you supporting tax documentation for workers? Collecting W9s, etc?
• Contractor agreements? NDA? What data do you need on us in general? Some workers like mTurk for its anonymity, but most folks will abandon that super quickly if the pay is worthwhile but its good info to put out there.

Just in general, all the baseline expectations a worker can expect to walk into when dealing with your platform. Folks here are absolutely amazing, and without question some of the best online workers you'll find pooled together in one spot so more details makes it easier to broadcast out your info to them from my POV, ++ helps them make individual judgement on the platform's viability on a personal level.

Ah no way, you're probably not wasting your time. Saving yourself 20% & increasing data quality is nothing to scoff at. But I feel like the distance between that and "supporting the worker community" is galaxy sized massive hahaha. Building an in-house platform is cool. I've directly helped/supported at least 3 companies do the same thing in various facilities over the course of the last year and idk how many more indirectly, so I'm not trying to jump down your throat for doing your thing. But building something to support even our small teeny tiny niche corner of the worker community would be an entirely different ballgame than that so I'm mostly trying to figure out which direction you're most likely to go here again so I can accurately FWD folks to & fro :p It'd be a little awkward if you actually only need a dozen or so folks & I plastered a banner at the top of the site promoting your platform for signups, y'know? Lol.

Please do! FWIW we have some incredibly skilled & experienced online workers, and most of them provide incredibly good data and feedback for what they earn day-to-day. On average folks here will be in the top 5% or so of workers for mTurk when put into comparison from research data from places like Pew. Moreover, we tend to solve our own problems with the platforms we use, which honestly is worth its weight in gold for someone building their own lol.

 

I was going to write something out.. yeah that sums it up.

I would be interested in at least checking it out as well if you need users. One thing that is very important is a way to notify workers when work goes up as we won't have our setups that alerts us to stuff like we have for mturk.

 

what everyone alrdy said in one shape or another

 

Wow, massively appreciate everyone's thoughts and enthusiasm about this. Thank you.

We need to properly formulate responses to all of the questions and feedback raised, but I wanted to focus on two things that seem key first:

• Pay & time. Clearly these are key for any professional endeavour. One of the challenges in this industry at the moment (I imagine - please correct me) is that many of the requesters are academics or startups (like us) who are looking for low cost, but workers are seeking more recognition and better value for the tangibles and intangibles they offer. We had some thoughts on this and would be interested in what you think:
  • Pay per unit time. Several people mention usability. If a platform enables workers to work much faster (via a Tinder-like swipe labelling interface for example) then workers are able to make more per hour even if the pay per label stays the same. Would this be attractive?
  • Larger committed jobs. If a worker could work on the basis of a committed number of tasks to complete - say 100k labels at $0.02 per label for a total job value of $2,000 guaranteed - would this be attractive/different from the current situation?
  • Value-based pay. If workers are labelling data for AI startups that go on to raise lots of money from successful algorithms based on labelled data, is there a way for workers to benefit from this? Is it worth exploring this kind of model?

• Supporting workers. As I mentioned before we're not an Mturk killer, an HIT platform isn't even our core business and we may yet find that Mturk is the best solution for us. But if we are going to try building our own platform we're keen to do it in a way that's positive for everyone involved. We can see how important HITs will be for us as an AI business, so for us this is about making small positive changes, even if only introducing ideas, that can help the worker community. It just makes sense that the arrangement between businesses and workers should be better structured and mutually beneficial if future business models (like AI) are so reliant on HIT workers. So I guess the question I'm asking is what are the changes we think the worker-startup industry needs to see at a macro level?

Regarding tangible next steps, we will share a link in this thread when our beta is ready but we'll probably only want a handful of workers to try it so we're not overwhelmed! So no need for further broadcasting at this stage please @ChrisTurk (but thank you)!

Cheers

Will

 

Personally the only reason that I think that the pay per unit is important on mturk is the hard cap of 3800 hits / day that we have. This most likely wouldn't be an issue on a new platform. At the end of the day we care about the hourly that we get while working on the task. If a task that is 50 cents each gives the same hourly as a $5 task it is more or less the same without a cap. It is worth considering load time on pages as for low paying tasks that can ruin the hourly.

As far as large guaranteed slots of work that we wouldn't have to fight over I believe that it would be attractive to most workers like myself as long as long as it is work that we want to do in the first place.

I'm sure that someone will come by and give a response that is better than what I just wrote out but this is more or less what I think.

 

pretty much just a repeat of what @Randomacts said it comes down to and hourly rate do the images load quickly, is the interface easy to work with or is it clunky with a bunch of unnecessary waiting, scrolling and clicking that kills the hourly. Also a guaranteed amount of work we do not have to fight over is always attractive.

 

Welcome to the forum. Some of this piggybacks on what Chris said. Although I'm blanking on names at the moment, there have definitely been platforms in the past centered around online advertising, and while I can remember a few being decent, I also remember that most, including the good ones, would disappear pretty quickly without much notice for a variety of reasons. This especially happened when Google and Facebook made changes to their search algorithms, some that had been around for years were just gone. So in addition to sharing more details on what others have pointed out, I'd encourage sharing the stability of the new platform with regards to external changes similar to the ones I described and what kind of sustainability and longevity you foresee for it in general.

 

Personally, I'm more comfortable with 'Pay per unit' since I have other things going at any given time in addition to my mturk work. Being able to sit down and hammer out 100, 500, or 1000 tasks depending on my free time nice. While @ChrisTurk may be hunting down all the hot dates on Tinder, I haven't ever messed with it, but a quick, swipeable interface could be interesting if you're intending on this being something that is to be done on smartphones. As far as interfaces go, a good portion of people here use scripts (some basic autohotkey, some javascript) to speed up their workflow, so if the work is being done on computer, I would think a simple interface with radio buttons or whatever may suffice. I'm not saying be lazy and build it as ugly and cheaply as possible, but when microtasks are the job, milliseconds count, and I don't really care about the interface as much as quick loading.

Larger committed jobs are attractive in the fact that it is guaranteed work once it has been assigned, but the timeframe for getting the work done and the quickness/ease of tasks would play a part in my decision.

Maybe a mix of the two. Pay per unit, then after a certain number of those have been completed, the ability to sign up for batch work. I wouldn't want to commit to a large batch without getting some tasks under my belt.

Value-based pay is an interesting idea. Could be implemented in a royalty like system where the worker pool gets X% royalty. Then the pool is divided up by the number of tasks completed.

 

Hello all - apologies for the lack of updates on this - we have been building the platform out as a side project so it's taken some time!

We are now running a closed beta and would love the team from this original thread to participate and feed back if you are still interested. For this beta there are 5 live tasks each with 5,000 items to be labelled at $0.01 USD each. The tasks involve scoring the creativity of short advertising copy lines. In tests so far users are able to do 1 item every 2s equating to roughly $18 USD/hour. Payment at the moment is via a Transferwise link sent to your registered e-mail address at the closure of the beta on 30th June, or when all 5 tasks have 5,000 aggregate labels across users. There is a live chat happening in parallel on Slack where you can interact with other workers and the developers if you wish.

If you would like to participate, please send me a DM with your @gmail.com address and I will send you an invitation.

Thanks very much for the feedback and support so far! If you have any suggestions or concerns please do let us know here.

Cheers

Will

 

I really like this interface so far.

 

I like that it works the same on my desktop and phone. I'm not usually a big one for doing hits on mobile devices but these are quick and simple enough to do even when I'm out running errands and find myself with a few minutes

 

Are you really getting quality judgments at that speed? I'm curious to check out the platform but I'm definitely not working for .01 tasks. Not worth my time.

 

actually for me atleast it has added up just doing as filler or in my spare time while waiting at restaurants or what ever over the weekend it is really fast and time will improve as the bugs are worked out and I'm not second guessing myself