best way to run attention checks?

Discussion in 'Requesters' started by oliviakim509, Jan 12, 2021.

?

Would you prefer an attention check where:

  1. There is more text on the screen, but the answer to the attention check is always visible.

    2 vote(s)
    33.3%
  2. The only text on the screen is the attention check question and the possible responses.

    4 vote(s)
    66.7%
  3. Other (would appreciate it if you could comment what you would prefer)

    0 vote(s)
    0.0%
  1. oliviakim509

    oliviakim509 New Turker

    Messages:
    7
    Gender:
    Female
    Ratings:
    +2
    Hi y'all! I am a university researcher who is pretty new to mTurk, and I am trying to improve my HITs as much as possible. I hear back from workers that I give too many attention checks, but I need to run these checks so that I can interpret my data later. I am hoping to make the attention checks less annoying, and was wondering what you guys thought would be better (poll above):
     
    Last edited: Jan 12, 2021
  2. Slowdive

    Slowdive Active Turker

    Messages:
    637
    Gender:
    Male
    Ratings:
    +1,549
    I think it sort of depends on the nature of the study and the type of attention checks you're using. Are they something like "Choose Option 3" or are they asking a specific question about a text?
     
  3. oliviakim509

    oliviakim509 New Turker

    Messages:
    7
    Gender:
    Female
    Ratings:
    +2
    They're asking a specific question about the task instructions. This is the current text of one attention check for an experiment where we ask people to try to move the mouse to a pink target when there is a misleading "fake" cursor on the screen:

    "Instructions and Attention Check

    The green dot is not under your control.
    It will still move at the same time as you, but it could go somewhere else.
    Do your best to IGNORE the green dot.
    Always aim for the pink target.

    Press x if you should ignore the green dot and aim straight for the pink target.
    Press y if you should aim away from the pink target to try to get the green dot to hit the target.

    [option to press x] [option to press y]"

    Right now, the instructions appear at the beginning of the HIT and during the attention check. I could take the instructions out of the attention check so that there's less reading to do during the HIT itself, and rephrase to something like:

    "Press x if you should move your mouse to the pink target and ignore the green dot.
    Press y if you should aim to hit the pink target with the green dot.

    [option to press x] [option to press y]"​
     
  4. Slowdive

    Slowdive Active Turker

    Messages:
    637
    Gender:
    Male
    Ratings:
    +1,549
    In that case I would leave the instructions on screen during the attention check.
     
    • Like Like x 1
  5. oliviakim509

    oliviakim509 New Turker

    Messages:
    7
    Gender:
    Female
    Ratings:
    +2
    Thanks!
     
    • Like Like x 1
  6. angel

    angel Survey Slinger

    Messages:
    15,286
    Gender:
    Female
    Ratings:
    +28,535
    As a worker I expect to get attention checks but they should be occasional not multiple checks per page. I have done surveys where I am certain there where more ACs than actual questions. Also the type can be an issue they should be of the nature that they will not be missed if you were at all paying attention. They should not be tricky or ambiguous so that workers are looking at them trying to guess what you were thinking when you wrote it and afraid to proceed.

    And my new pet peeve DO NOT put AC's in consent forms most workers scan through them but do not read every word after all they are largely all the same with slight differences if there is something different about that research project (thus the scanning). Lately putting some secret phrase in the middle of a consent form and asking about it later seems to be a popular thing and is very frustrating to workers.

    Beyond that what are you currently using. Do you feel you need something beyond the standard if your paying attention select option 3t type AC. Or the research shows decisions aren't made.... so select ...
     
    • Like Like x 4
    Last edited: Jan 12, 2021
  7. angel

    angel Survey Slinger

    Messages:
    15,286
    Gender:
    Female
    Ratings:
    +28,535
    just noticed the example above. It is wordy but I agree with @Slowdive I would leave it in. Is this AC occurring during the activity if so workers might find it less annoying and less distracting if you gave instructions then a brief quiz on the instructions this is fairly common when making sure participants understand the instructions is important.
     
    Last edited: Jan 12, 2021
  8. oliviakim509

    oliviakim509 New Turker

    Messages:
    7
    Gender:
    Female
    Ratings:
    +2
    Ahh... I think that this is a possible attention check format because it's really easy to program/write in. It's really good to learn that this is frustrating and something to avoid.
    Now that I think about it, these hidden-phrase types of attention checks are not that useful anyway. They ask the worker has to remember some random phrase instead of the actual instruction or whatever thing the requester wanted to make sure is understood.




    Thanks for your feedback! I will try to cut down the words/make it clearer.
    This attention check occurs during the activity at a point where the instructions change a little bit. There are new instructions every minute or so for the first 4 minutes, and then things are consistent for the 5 or 6 minutes after that, so there are 4 attention checks in total but there are a bunch in a few minutes. There is a common goal throughout the whole HIT, though, so maybe I can cut down to 2 instructions quizzes -- one at the very beginning and one at the end.
     
    • Like Like x 1