Attention and Manipulation Checks

Attention checks, comprehension checks, and manipulation checks help researchers understand whether respondents read instructions, understood a treatment, or experienced the intended experimental manipulation. They should answer a specific research or quality-control question, not simply add a hurdle because the survey feels important.

Plan each check before launch: where it appears, whether a failed check routes the respondent out, whether the result is an exclusion rule, and which export variable analysts should use.

• Attention checks: confirm respondents are reading instructions or options carefully, such as selecting a named option in a low-stakes item.
• Comprehension checks: confirm respondents understand a task, policy, vignette, or treatment before they answer outcome questions.
• Manipulation checks: measure whether the treatment changed the intended perception, belief, or awareness.
• Validation checks: make sure numeric or text answers follow the expected format, usually through required questions and validation rules.

1. Step 1: Place the check near the task it evaluates. Put a comprehension check after instructions or a stimulus, and put an attention check where it will not reveal the treatment logic.

   Step 1: Place the Check

2. Step 2: Add the check in Block Builder. Use Multiple Choice for closed checks, Short Answer with validation for typed checks, and Grid Matrix for repeated comprehension items.

   Step 2: Add Check in Block Builder

3. Step 3: Name the variable. Give the check a clear question name, such as attention_check, comprehension_policy, or manipulation_check, so it is easy to find in Export.

   Step 3: Name the Variable

4. Step 4: Route only when needed. If the answer determines whether respondents continue, open Flow and branch from the block that contains the check.

   Step 4: Route via Flow

5. Step 5: Preview both outcomes. In Preview, submit one passing response and one failing response to confirm the paths and exports.

   Step 5: Preview Both Outcomes

• Exclusion check: use only when the protocol says a failed answer should remove the respondent from analysis or collection.
• Diagnostic check: keep the respondent in the survey, export the variable, and decide during analysis how to use it.
• Comprehension retry: route a failed respondent to an instruction reminder, then let them answer again if the protocol allows retries.
• Manipulation measure: avoid using it as an attention check if it is substantively part of the outcome model.

Decide before launch whether checks are exclusion criteria, diagnostic variables, or outcomes. Document that decision in a pre-analysis plan or codebook so the export is interpreted consistently. If a check changes after launch, record the date, old wording, new wording, and affected response ids.

• Survey Data Quality Checks
• Online Survey Experiments
• Multiple Choice Survey Questions
• Short Answer Survey Questions
• Pre-Analysis Plans and Survey Codebooks