I did it - will post to Daily Work Thread. I will monitor this for issues - today is my son's bday and so I could be periodically distracted. Thanks and good luck
So I'm looking over the HITs, and to me, it looks like the ToC data should never be used as line numbers. I haven't done any yet, but I will if you'd let me know when exactly it's appropriate to use ToC data? Because from the looks of it, there's almost always an Item 6 or Item 7, even if it's linking to another document, it'll describe the location in the body of the text rather than the ToC.
Ooops, included the entire **0000XXX** in my first ones. What do you want us to return if that section is not included? I had one where it simply referenced a section of the annual report and another that was a completely different type of report.
Doing one now where I found where item 7 starts and item 9 starts am I correct there must be an item 8 in between that I'm missing and which would be the second number your looking for.
There are cases when the reference to the other document is in the TOC please see the examples as I have one there
The line numbers where it does reference the annual report. I have an example of something possibly similar in the instructions
that's what I ended up doing, went through that section several times and if there was and item 8 then I wan't seeing it LOL
I have just finished the first step of the approval process and approved 1,551 of the initial batch of 2,063. The mTurk console is chewing through my file right now and should notify you soon. I also approved the $0.10 bonus on 773 of these 1,551 but I am still a little fuzzy on how to do this in a batch made through the console so you may not see the bonus until I can sort that out and I have to stop work for the day. I want to note that I have only rejected one so far. I need to look at the remaining a little more carefully to decide to approve or reject. So if you have some outstanding do not imagine that they have been rejected - some I know I will approve but I need to sort out the approval process so I can do it systematically and recover what I need from the files. Some of them will help me enhance my instructions but unfortunately some are just going to be rejects. My guess right now is that about 25% of the remaining 500 or so will be rejected. The ones I will not reject will be because I think you tried to get it right but my instructions did not anticipate those cases. I will note that I did segregate the cases where you indicated that the content I was looking for was not in the file (those are in the remaining ones). If you are correct you will get the base pay plus the $0.50 bonus I noted yesterday. I hope to have the rest of this done before Monday afternoon but I gotta spend some time with my favorite son. Thanks for your patience.
You should be careful with rejections as a tool. My honest suggestion is that unless a worker is consistently submitting bad quality work there isn't any reason to reject one off HITs. There are a huge amount of workers who will avoid you like the plague if there are rejection reports on your TO because 1) rejections are disproportionately damaging to a worker's account (it takes 1,000 HITs for every single rejection to "make up" for the rejected HIT) and 2) it tends to look a bit nit-picky. Part of the downside of a platform which uses humans is that human error is a real thing. Making a mistake and having payment denied and reputation damaged tends to be looked at as bad faith on the requester's part. Imagine if your boss docked you every time you sneezed during a presentation or what have you. Some folks don't mind it too terribly (I don't if the work pays very well and its not subjective) but I'd brave saying that the majority of workers wont work for requesters who reject sporadically or without incredibly good logic/feedback. Admittedly even I get unreasonably salty when A9 rejects a random one-off HIT I make a mistake on, it just feels very bad to work for someone (especially if its high volume) and have one-off HITs rejected. Its usually not about the pay, but the humanity/relationship element between requester/worker - so its hard to overcome with logic sometimes. Hope that makes sense / is helpful haha. Now by all means, screen out poor quality workers (there are two good but different ways to do this I'll outline below). And if you truly believe a worker is attempting to scam you or is submitting work in bad faith (clearly not reading directions on a majority of HITs, etc etc) - reject away because those are the workers who ruin it for everybody. Screening out poor quality workers If you still want a large worker pool (best if you have a time crunch on your batches and can sacrifice a bit of quality for speed): Create a qual w/ a simple name like "Excluded" or something similar Assign it to workers who do not meet your quality standards Add a qualification to your HITs "Excluded has not been granted" From here on out anyone w/ that qual wont be able to complete your HITs, no damage has been done to their overall account (some people aren't good at certain tasks, even if they try, but its best to assume good faith and let other requesters deal with them as they see fit than to harm your own reputation) "Hiring" a high quality pool of workers If data quality is far more important than the speed your batches get done you can create a "closed qual" for workers who submitted high quality data on your batch. Create a qual w/ a name like "Good worker" or w/e you prefer Assign it to workers who do meet your quality standards. Add the qualification to your HITs as "Good worker has been granted" And then only those workers will be able to do them. You can recruit more workers through any means you deem fit: test batches, word of mouth, qualification tests, the options are limitless and you can pick what works best for you.
Humm this is interesting I understand the observation and sort of get it (in my day job I am on the faculty at a university). I was initially going to approve every single hit and then deal with the consequences myself with the notion that any errors are my fault - bad examples or incomplete. Having said that I have already noticed that one person worked one of the examples and they did it wrong - exactly the way I said not to do it. I want to look at their overall rate. Some of the other bad hits appear to be the case where the person did not read the instructions. I am going through these with a very open mind and am trying to sort out bad communication or some other issue. I do suspect that the vast majority of these are going to approved. There is a lot to learn here and I am going to do the best I can
I finally sorted out how to pay a bonus using boto and so I have paid a bonus to 737 assignments where the returned line span was greater than 250 (I actually did span > 240) so that should show up in your accounts.
So I am struggling here with you comment above. I am checking the ones that have to be hand checked right now and for this item the person who took this task placed the starting line based on this image They gave me line 2558 as the starting line The correct starting line was 1582 Is there something I am missing about this that should stop me from rejecting this item?
On the other hand - this one was not what I was looking for - the information indicates that the required disclosure is a bit further down in the document but I think this is my fault and so I am approving this one and will modify the instructions to describe this case
Looks like a silly mistake. The person probably hit "ctrl+f" and searched for "Item 7" or something similar and his browser took him to that line instead of the first instance for whatever reason. Alternatively, he could have started by searching for Item 8 to get the last line then just scrolled up until he saw Item 7, didn't notice that it says "conclusion" and just assumed that was correct. Whether you reject work like this or not is up to you. But examples like this go back to the "human element" of mTurk. You don't have a computer searching the document in a super standard way, you've got dozens of humans with all kinds of setups/browsers/scripts/work methods/thoughts on the most efficient way to complete work. Many of them probably have absolutely no clue what they're even looking at, so where you may see "Concluded" (or any of a thousand flags that stick out in your head because you're intimately familiar with the project) and instantly know that's a mistake it may not even blip on their radar. I'd be surprised if this person purposefully did this incorrectly out of malice or laziness since it seems like there are a multitude of ways this kind of mistake could happen depending on their workflow / method of searching the document / etc. Again, its up to you, but that's a mistake I easily could have made working on them haha.
I think the overall point is that requesters who reject for all mistakes will be avoided by many workers. As Chris said, rejections are harder to recover from than just about anything else on mturk, rejections do not just affect our ability to do your work, it affects our ability to do all work. Rejecting bad work is not wrong, rejecting work from a worker that consistently does good work for you and messed up on an item or two after completing ~500 of your batch is what will make people avoid you. Closed quals seem to be the best way to ensure quality if you are really seeking that. I'm not the most experienced turker but a good method could simply be setting up a qual that, "requires a test." to acquire. The previous instruction page you linked to could be formatted back into a test that they have to pass in order to get the qualification. So if they failed to come to the answer, starting line 767 and end line 1147 they would fail to get the qual.
So I am still struggling here but went ahead and 'APPROVED' the remaining 512 hits that I am having to manually verify. I have made a ton of progress on those but I don't want to work on this anymore over this weekend and even to consolidate everything and prepare it for submission would take more time then I have today. I don't want those of you that did a great job to wait for at least the initial payment. The down side of it is that there are probably 200 +/- hits where the Worker deserves a $0.10 bonus but I can't pay that bonus until I do finish the hand validation. Also there are about 30 hits where the Worker deserves the trues $0.50 bonus because the content was legitimately not present. The bad thing is that there are also about 30 hits where the worker claimed N/A but the content was exactly in the documents in a manner that was indicated by one of the examples. I have a lot of analysis that I need to finish up before I move forward. There were some folks who just did an amazingly good job. There were some folks who appear to have done of few without much care and then moved on. For those of you that are still owed a bonus I hope you can be patient with me. Part of it is my fault as while I imagined the steps I would take to validate the results it is more tedious than I imagined.
Wow I had a brain blast - it has been painful to check these and I have been mulling things over a bit and a little bit ago wondered why can't I use the Mturk Interface to check them - so I decided to try that - using the sandbox and it is amazingly fast. Much more efficient then using the tools I have been using. The first batch I approved I was able to automatically check but these remaining require me to look at the documents and look at what was submitted. I had been worried about going forward and managing this but I think this is going to work well. I just checked 50 hits in under 30 minutes.
I just released the NA Bonus I added after the job started. If you did not get one that means I found the content that you indicated was not present and I believed you should have found it.