Maintaining Survey Response Quality

Response quality is not a single setting. It comes from the survey design, the respondent source, pilot testing, live monitoring, and the way exclusions are documented for analysis. Domandata helps you plan those checks before launch and carry the review record into exports, replication packages, notebooks, and API pulls.

Use this guide as the broad quality pass for academic and applied research surveys. For the focused tools in the Export tab, see Quality Review Queue and Cleaning Log and Author-Defined Response Quality Rules.

1. Step 1: Write quality criteria before data collection. Decide which checks are exclusions, which are diagnostic flags, and which are substantive measures.

   Step 1: Plan Criteria

2. Step 2: Add checks where they belong. Put screeners, validation rules, attention checks, comprehension checks, and manipulation checks near the task they evaluate.

   Step 2: Add Checks

3. Step 3: Test every route. Preview eligible, ineligible, quota-full, treatment, control, pass-check, and fail-check paths.

   Step 3: Test Routes

4. Step 4: Confirm the export shape. Submit pilot responses and make sure response ids, source fields, quality flags, condition assignments, variables, and randomized order fields appear where expected.

   Step 4: Inspect Export

5. Step 5: Configure author-defined rules. Add explicit correct answers, impossible combinations, dwell-time thresholds, and source defaults in the Quality Review panel before launch when possible.

   Step 5: Configure Rules

• Attention and manipulation checks: answers that should match an explicit correct value or expected treatment interpretation.
• Impossible combinations: pairs or sets of answers that should not occur together, such as mutually exclusive screener choices.
• Timing: total completion time, page dwell time, and author-defined minimum dwell expectations.
• Duplicate tokens: repeated device or browser identifiers when the available metadata indicates likely duplication.
• Bot signals: reCAPTCHA score and pass/fail metadata when reCAPTCHA is enabled.
• Open-text quality: repeated or unusually low-effort text patterns.
• Missingness: excessive missing answers across answer-bearing questions.
• Source defaults: review decisions or tags applied by recruitment source, such as pilot links, panel batches, or known test channels.

Automated signals should start review, not end it. The final include or exclude decision should follow the protocol, the cleaning log, and any pre-analysis plan.

• Check response counts, source labels, quotas, and completion patterns after the first live responses arrive.
• Use the quality queue to review flagged responses without deleting the raw row.
• Mark clear test submissions, failed checks, or invalid sources with include/exclude tags rather than editing answers.
• Watch source-level patterns, especially when using panel vendors or multiple recruitment channels.
• Download backup exports before changing wording, Flow, quotas, links, or end screens mid-field.

Use Export as the bridge from raw responses to analysis. Domandata keeps quality metadata with each response and adds a cleaning log to replication packages. The analysis templates load the cleaning log and create analysis-ready data frames that exclude rows marked exclude.

• Keep the raw response export unchanged.
• Use Quality Cleaning Action to create the analytic dataset.
• Review Quality Rule Matches before treating a flag as an exclusion.
• Preserve Quality Review Source so analysts know whether a decision was manual or came from a source default.
• Archive the cleaning log with the codebook, survey instrument, and replication package.

• Author-Defined Response Quality Rules
• Quality Review Queue and Cleaning Log
• Attention and Manipulation Checks
• Review and Clean Survey Responses
• Export Research Survey Responses