Bonus Structure

Hey guys,

We're working on developing our pay structure and one of the things that makes sense to us is the idea of including a bonus when fires are detected.

It's a big deal to find a smoke. Early detection plays an incredible role in successful fire suppression which in turn has a profound impact on the environment, communities, and individuals that fire affects. Providing bonuses to people that find them, makes sense.

However, we're struggling a little. We're worried about people gaming the system. Most of the time there won't be smoke and so it will be really easy for people to fly through our hotkeys. Even the most honest workers (like @rater9 ), after two months of not ever seeing a smoke, might grow a little weary and pay less attention than they were at the beginning of the trial.

So to combat this, our plan is to litter our data set with fake smokes. Every now and then, to make sure people are keeping their eyes open, we'll place a false smoke in the HIT. It will either be from an existing data or an image that's photo shopped. But this creates a problem if we want to provide bonuses, we want to provide bonuses for real smoke, not fake ones.

So, we have two solutions and your feedback would be greatly appreciated.

1) We provide a small bonus every time smoke is correctly identified. So if each image is $ .01 , then a positive smoke ID might be $ .02 (Initially we were going to provide $.10 - .15 bonuses for positive smoke IDs, but now, with a plan to have a lot more, that's not really feasible).

2) We provide a $10 bonus once a Turker has successfully analysed something like 1000 - 2000 consecutive Hummingbird images (these numbers are made up right now, I have to figure out the actual numbers). This option is my preference. It will be a higher bonus, encourage people to be accurate in every photo whether or not there is smoke.

I've created a poll below.

We really appreciate your feedback. If there are alternatives, let us know!

R

Edit 1: The poll is actually above this post.

Edit 2: Also, if you haven't had a chance, check out hummingbird.watch for our HIT preview

Edit 3: It might be too late to gain any traction - but you will still get paid for every HIT. The bonus is on top of that!

 

Absolutely do not bonus positive smoke ids. You are just asking for way too many false positives.

On the other hand, you certainly want to be biased towards false positives and not false negatives.

 

Just to clarify, individual $.02 per image would be rare. I'm guessing right now, but maybe one out of every 20 images. Whereas if you banked and were rewarded something like $10 after 1000 images, you would actually be doubling the income from each correctly ID image.

 

We actually don't mind false positives. We are far, far more worried about missing true positives.

Edit: Within reason of course, we don't want false positives in every HIT. Definitely worth considering.

 

Obviously we would be biased to more mula. However, I would think the number of people that hit 1000 images would be relatively few. My point was and is, the first bonus structure will biasse your results towards positives.

 

Agreed. Our conversations internally have focused a lot on the potential of people becoming "bored" with the tasks.

 

Turkers have mastered the art of being bored and doing tasks.....My understanding is they will only be posted every 5-15 minutes, so you will have people running PANDAs to catch them. This isn't like GeoHive or something where you are literally staring at 1000s of images in a few hours.

 

Haha, fair. Yea, so HITs will be posted every five minutes, each HIT (initially) will consist of 5 images and each image will be worth $.01

Edit: So maybe 1 image every 4 or 5 HITs will have a smoke in it.

 

So with option 1, you would be earning a bonus at maybe a ratio of 1 / 25

Whereas option 2, your bonus would be closer to a 1 / 2 or 1 / 1 ratio. It just wouldn't be immediate.

 

I also doubt the same people will be doing every hit. Not sure how many samples you will take, but if 20 people are running a panda and you only post for 3-5, one person may only do the hit 2-3x an hour.

 

I see boredom not being an issue here. 5 images every 5 minutes is nothing. If you have a worker account, look at the DGIS post now. Batch of 2000 HITS, each having 50 images. I mean, there you have to wonder if people don't enter a trance. I'd be curious what their data quality is at the end of a batch vs the beginning.

 

Yea, but that kind of works both ways. It will also drastically reduce the probability of finding a smoke if it's a per image thing.

Your picture is hilarious by the way

 

That's one of our concerns. High quality image analysis is incredibly important here.

 

Haha, the poll is indicating that I'm not doing a great job of selling option 2.

 

Thanks @gurlondrums

We're excited to get things rolling too.

I wonder, have I done a good job explaining that bonuses are on-top of regular payment? People will still get paid immediately.

 

We're also now at a 50/50 split in the poll. We'll have to see what replies look like over the course of the week.