Varying difficulty among HITs (how to generate and pay?)

Hello everyone!



I have an interface where every different HIT ideally will have different number of text boxes to fill in. I know that I can prepare such a thing with JS but how do I pay different for each HIT? Ideally I would like to pay x * no.of textboxes. Is it even possible? What do people do it such settings?



Also, let me know if there's a better to way to have different HTMLs for different HITs, will be super helpful!



Sincerely,

Sumith

 

Some requesters with similar situations just have a base pay so they can put out a whole batch, and they give a bonus on the hits that have more work in them.

 

I was of the opinion that turkers might find it frustrating that every other HIT doesn't justify the pay (the variance can be 2-4x), I do not want to put people in inconvenience. Any thoughts about that? What is the general crowd opinion about having lots of bonus with some base pay (almost 90% might be bonus)?

 

Do you have any examples of the HITs already? The thought of using JS to generate the task is scary at face value but maybe I'm not understanding the setup here.

This is the simpler of two options and it's pretty effective if you build a reputation first since workers wont mind as long as you bonus quickly & fairly

 

I've seen at least a dozen requesters do it and have no problems at all. I wouldn't say its common (most tasks don't need to be so variable) but it's a well understood process among experienced workers

 

Not yet with JS but this a sample of my survey https://cs.stanford.edu/~sumith/mturk-qual.html

You can find the instruction set I'm planning to use here https://docs.google.com/document/d/1_XuQRb679Ny2Qs5c17zL5uZr4QqEazhkxpBDOjKJI0U/edit?usp=sharing

The deal is to annotate each of program with a simple description. I plan on having different HITS as different programs which can have varying lines of code and hence varying textareas.

Awesome. I guess the account I'm using is associated with the university and has a fair reputation which will come handy. The payments will be out in time too!

 

Oh wow, honestly I feel like you are much more likely to have problems with this. 7 pages of instructions, on a highly complicated topic, is going to be difficult for a lot of workers to get through. Is this going to be some kind of paid qualification task or are you hoping to get folks already familiar w/ the language? If the former, this is far too verbose/complicated.

Going off the preview task, at 40c/HIT you'd need to do 25 HITs per hour to make $10/hr.. that's 2.5min/HIT for 20 LOC? How do the actual HITs compare to the qual test?

The qual test took me 3min, 12s for 12 questions (not counting time for glancing at the instructions document before starting), and the 12th I put NA for. If the actual HITs have 8 more LOC this is already a bad storm brewing. I'm familiar with a handful of programming languages, though self taught & not by trade, so I'm already skewing low for completion time vs an average worker if I had to guess.

This is going to be abused, and you are not going to get usable data from it, IMO.

 

Hi Chris,

Thanks a lot for glancing over the doc for me! Super helpful.

We did a paid qualification round (the survey you saw) and are planning to only go forward with the turkers performing well on that. The instructions are plenty of examples coupled 2 pages of things not to do. I tried to go over the board with this maybe but I wanted people to have clarity on what needs to be done.

I agree that the pay shown is on the lower side, I'm happy to fix it. I can maybe try to request for $0.60 for 20 fields (1.5x current rate) and if approved would be much better. (In my guess $10 per hour for decent pace and over that for better pace and the task gets much quicker after the first few HITs). Thoughts about that?

Sincerely,
Sumith

 

Oh, that's awesome then haha. I was questionable on reading all that for an unpaid qual hit, but that sounds good long as y'all grabbed some decent folks

The other thing you can look to do is increase the efficiency of the task itself. You can't always increase the payment, but you can definitely do things for the layout/flow of the work to make it a better experience.

Scrollbars are the devil for workers. Any time spent touching a mouse is an efficiency loss, for the most part. So while that example form is pretty, at the end of the day most of us are here to work not look at empty white space & CSS

There are definitely ways of going too far the other direction, but as-is throwing something together like integrating CodeMirror & condensing the page or reformatting the layout would be incredibly helpful from a workflow perspective which will cut down on time spent working (or not working as you bounce to new questions / try to parse syntax / etc)

A couple of hours on your end for reevaluating the task can probably squeeze out a good bit of efficiency gains that'll be amplified by however many tasks you end up sending out to workers

 

Hi Chris,

I'll take the efficiency factor into consideration and restructure the task, (CodeMirror looks amazing, will see if it is possible to have that!) I might reach out again once that's in place. Thanks a lot for your time!

Sincerely,
Sumith

 

The base pay and bonus for more work is probably your best bet just make it very clear in the instruction what the bonus is x amount per text box for example so that workers are not looking at a hit with a bunch of text boxes and thinking it is underpaid.

My personal prefrenceas a worker is if this allows you to put out a batch it is a better experience for the worker than having a bunch of different batches with 1 or some other very small number of hits that spam the site and have to be caught individually

 

I know this is offtopic, but I really enjoyed doing the qualifier. I hope you end up posting a bunch of these at a reasonable wage. One problem I had in doing the qualifier was I wasn't exactly sure who I should be describing the code for. Someone who knows nothing? Someone who codes, but doesn't know C? Or? Your examples kind of run the gamut of that. If you clarify that when you publish your hits I would be able to proceed with a lot more confidence.

 

Hello all,

We have now posted new batches with automatic qualifications (plus all the feedback got here earlier incorporated!). Do let me know what you think of it. Also if I need help to advertise this task and interact with the workers performing (or not performing for whatever reason) this task, what would be the best route?

Sincerely,
Sumith

 

these hits probably aren't my thing but if you want to advertise them post on the daily thread with a link back to this thread and maybe a link to the hits or any qualification hit you have set up

 

Thanks a lot @angel , will do