Hi Everyone, We are new requesters AVID annotation and we’d like to get any feedback and guidance we can from the Workers here. Previously we’ve put out smaller batches of HITs to experiment, but we weren’t getting the response we had hoped for. Now we are hoping to improve on that by putting out a larger batch of HITs at once and also engaging with the Worker community. We definitely want to create worthwhile HITs and are willing to make changes. Please respond here or feel free to send us feedback at avid-annotation@kitware.com. We are planning to create a large number of hits over time, and would want to have repeated workers who do well at the task and enjoy the HITs. Our HITs are intended to annotate objects in videos, related to specific activities. The activity name will be included in the UI. These are videos where a specific activity will occur, and a box is drawn around the objects related to the video at the start and the end of the activity. The Worker’s job is to correct the annotation boxes throughout the video so that the box is somewhat precisely over the object to be annotated. Here are a few example HITs that are in a preview/development mode (they are safe to play with, but won't pay any reward) if anyone wants to check them out. Simple action: sitting down Two people involved: people talking A person and an object: picking up an object The below pictures are included in our instructions. We have put out an initial batch of about 100 HITs, each paying 20 cents. Is this a good value, or is there something we can do to make the HITs more appealing, or to make the HIT more understandable? The HITs themselves will vary in difficulty and hence expected time, this is something we hope to get a better understanding of so we can potentially choose the right reward for different HITs. In later batches we would like to use the qualifications, but for we are trying to get some initial responses. As we are new to using qualifications, if anyone has advice there or about how to interact with the Worker community please let us know. Also advice on our UI or instructions would be appreciated. Thanks, The Avid Annotation Team
I don't know how long the videos are but I bet that most people noped out after seeing bounding boxes + videos as those are almost always underpaid. How long does each hit take to do?
Only major request I'd have from a 10 second UI overview is please let me resize the box by the sides alone. Corner drags tend to let me fix one side while simultaneously screwing up the other. Also the ability to move the box entirely w/o resizing would be nice but perhaps I just missed that. I've gotta be honest though - I've never seen one of these tasks be worth it for me personally. They just tend to be one of those things that end up outsourced to a non-US/UK/CA country because it requires next to no real skill & certainly not much beyond a very basic understanding of the English language to get through the instructions. With that said I've never run one of these projects so maybe Americans have some innate skill required to do them well, that's just my personal musing on the pay rate dilemma you're gonna face coming to a site like this for advice in that area At the end of the day whats right / works for your business is up to y'all, just know this particular worker base is going to trend toward higher wages than mTurk in general because, well most of us have better things to do than these kinds of HITs haha. Your video example was about a minute long and seemed to be the best case scenario for these tasks. Even at that rate, I'd honestly rather do pretty much anything else than one of these kinds of HITs I tried fiddling with the old guy w/ a backpack task and personally couldn't envision a scenario where that would ever come close to even minimum wage. There are plenty of workers on mTurk who can/will, though, so I'm sure y'all will be fine overall.
Hey there. We always appreciate when a requester comes on here and engages in the community, so thank you first off! Second, like Randomacts said, how long does each HIT take for a worker to complete. You're paying 20 cents per HIT so if they're taking any longer than about 2 minutes to complete the hourly rate dips below minimum wage. We'd at least like to get paid $8-10/hour if you want quality workers to take a look at your stuff.
Thanks for coming here and discussing your hits with us! I don't see any hits up right now to actually do and see how long it takes but the links didn't look too bad. I don't hate bounding box stuff if it's safe and I can make at least minimum wage on it. I'd say $8-$10/hr would keep me relatively involved working on the batch. Using quals will help for safety, if you don't think we are doing a good job you can pull the qual instead if handing out a bunch of rejections. Length of video will effect what the pay should be and you should be paying more for the ones that have you annotate multiple objects because it will take longer.
I agree with what @ChrisTurk turk said about resizing the boxes. I am not inherintly oppossed to this type of hit but so many want the boxes as tight as possible but make it very clunky and difficut to adjust the boxes so that it becomes more trouble than it is worth. That said there was one requester a whilt back that also did video annotations and the boxes were easy to work with and so the hits ended up being worthwhile. The other issue as many have said is the length of the videos and how many boxes nedd done per hit wich boh effect the time needed per hit and thus the hourly pay
