Maintaining Survey Response Quality

Response quality is not a single setting. It comes from the survey design, the respondent source, pilot testing, live monitoring, and the way exclusions are documented for analysis. Domandata helps you plan those checks before launch and carry the review record into exports, replication packages, notebooks, and API pulls.

Use this guide as the broad quality pass for academic and applied research surveys. For the focused tools in the Export tab, see Quality Review Queue and Cleaning Log and Author-Defined Response Quality Rules.

1. Step 1: Write quality criteria before data collection. Decide which checks are exclusions, which are diagnostic flags, and which are substantive measures.

   Step 1: Plan Criteria

2. Step 2: Add checks where they belong. Put screeners, validation rules, attention checks, comprehension checks, and manipulation checks near the task they evaluate.

   Step 2: Add Checks

3. Step 3: Test every route. Preview eligible, ineligible, quota-full, treatment, control, pass-check, and fail-check paths.

   Step 3: Test Routes

4. Step 4: Confirm the export shape. Submit pilot responses and make sure response ids, source fields, quality flags, condition assignments, variables, and randomized order fields appear where expected.

   Step 4: Inspect Export

5. Step 5: Configure author-defined rules. Add explicit correct answers, impossible combinations, dwell-time thresholds, and source defaults in the Quality Review panel before launch when possible.

   Step 5: Configure Rules

• Attention and manipulation checks: answers that should match an explicit correct value or expected treatment interpretation.
• Impossible combinations: pairs or sets of answers that should not occur together, such as mutually exclusive screener choices.
• Timing: total completion time, page dwell time, and author-defined minimum dwell expectations.
• Duplicate tokens: repeated device or browser identifiers when the available metadata indicates likely duplication.
• Bot signals: reCAPTCHA score and pass/fail metadata when reCAPTCHA is enabled.
• VPN/proxy detection: an opt-in check (Survey Options → Fraud Detection) flagging responses submitted from a known VPN, proxy, Tor, or hosting-network IP. Respondents in the EU, EEA, UK, and Switzerland see a short notice that their IP address is checked with an outside service.
• Open-text quality: repeated or unusually low-effort text patterns. The gibberish and duplicate-text checks also cover typed side fields: "Other" text on choice questions, custom Card Sort category names, and Text Highlighter notes.
• Typing and pasting: opt-in typing timing (keystrokes, typing duration, longest pause) on Short Answer and Text Chat, and paste tracking or blocking on open-text boxes, including Text Highlighter notes and the typed-answer box on audio questions.
• Missingness: excessive missing answers across answer-bearing questions.
• Source defaults: review decisions or tags applied by recruitment source, such as pilot links, panel batches, or known test channels — including a panel-supplied numeric field (e.g. an approval rate passed in through the recruitment-platform URL) falling below a threshold you set.

Each flag names the question by its export variable name: the custom question name if you set one, otherwise its column label in the CSV (Ask1, Ask2, and so on).

Automated signals should start review, not end it. The final include or exclude decision should follow the protocol, the cleaning log, and any pre-analysis plan.

• Check response counts, source labels, quotas, and completion patterns after the first live responses arrive.
• Use the quality queue to review flagged responses without deleting the raw row.
• Mark clear test submissions, failed checks, or invalid sources with include/exclude tags rather than editing answers.
• Watch source-level patterns, especially when using panel vendors or multiple recruitment channels.
• Download backup exports before changing wording, Flow, quotas, links, or end screens mid-field.

If Domandata is connected to ChatGPT, you can ask ChatGPT to watch for flagged responses as they arrive, for example to tell you when a new response is flagged for speeding, straightlining, or a failed attention check. It is told which checks a response tripped and how serious they are, not the respondent's answers. Checks that compare a response against the whole sample, such as duplicate devices or duplicate open-text answers, are only run when you review responses in Export, so ChatGPT isn't told about those. This needs the connection to be allowed to access survey responses in Settings → AI.

Use Export as the bridge from raw responses to analysis. Domandata keeps quality metadata with each response and adds a cleaning log to replication packages. The analysis templates load the cleaning log and create analysis-ready data frames that exclude rows marked exclude.

• Keep the raw response export unchanged.
• Use Quality Cleaning Action to create the analytic dataset.
• Review Quality Rule Matches before treating a flag as an exclusion.
• Preserve Quality Review Source so analysts know whether a decision was manual or came from a source default.
• Archive the cleaning log with the codebook, survey instrument, and replication package.

• Author-Defined Response Quality Rules
• Quality Review Queue and Cleaning Log
• Attention and Manipulation Checks
• Review and Clean Survey Responses
• Export Research Survey Responses