All Research Guides

A Guide From Domandata

Bot and Fraud Detection for Online Surveys

Written by Domandata, which makes a survey platform. We name our product where it fits and say where another tool fits better. Claims about other tools link to their own documentation, checked ; confirm anything decisive with the vendor or your institution.

Fraud in online surveys is no longer occasional. Pew Research Center found that bogus respondents made up several percent of opt-in samples and, crucially, that most of them passed a standard attention check. More recent work shows that an autonomous AI agent can complete surveys while passing attention checks and common bot tests.

No single check solves this. What works is a set of layers, each catching what the others miss, plus a documented, preregistered way of deciding what to exclude. This guide walks through the layers and the tools that help at each one.

The short answer

Layer your defenses: recruit from a source that vets participants (Prolific, CloudResearch Connect, or a panel with pre-survey screening such as Research Defender), turn on the technical signals your survey platform offers (bot scores, duplicate detection, VPN checks), build checks into the instrument (attention items, impossible combinations, paste tracking on open ends, honeypots), and review flags against exclusion rules you preregistered. Where fraud would invalidate the study, use a probability panel.

The layers of defense

  1. 1

    Who gets in: the recruitment source

    Participant pools that verify identity and monitor behavior across studies stop many bad actors before your survey sees them. Aggregated panels vary more, so pair them with pre-survey screening.

  2. 2

    Technical signals

    Bot scores such as Google reCAPTCHA v3 (0.0 to 1.0, with 0.5 as Google's suggested starting threshold), device and browser duplicate detection, private-browsing detection, and VPN or proxy checks. Record these as data instead of silently blocking, so you can audit decisions later.

  3. 3

    Checks built into the instrument

    Attention and instructed-response items, impossible answer combinations, minimum reading time on key pages, and honeypot questions that people cannot see but scripts may answer. Pew found most bogus respondents passed a simple attention trap, so never rely on one check.

  4. 4

    Open-ended answers

    AI-written text is the newest threat. Track paste events and tab switching, set reasonable length limits, and read a sample of open ends yourself. Detection scores are likelihoods, not verdicts.

  5. 5

    Review and documentation

    Preregister exclusion criteria, review flagged responses against them, keep the raw data, and export a cleaning log. Reviewers increasingly ask how fraud was handled, and an auditable log answers them.

  6. 6

    When to pay for a probability sample

    If a few percent of fabricated responses could change your estimate, as with prevalence of rare behaviors, an address-recruited probability panel is the most robust option.

Tools at each layer

Vetting before the survey

Prolific verifies identity and offers optional in-study authenticity checks for bots and AI-written answers. CloudResearch vets Connect participants with its Sentry screening at sign-up. Research Defender scores respondents from any panel before they enter.

Prolific

Self-serve online participant pool for academic and commercial research, with participants mostly in OECD countries.

Pricing model
Pay as you go: participant rewards plus a percentage platform fee, lower for academic and non-profit accounts with an institutional email. No monthly fee.
Where it runs
Recruitment only; links to a study built in any survey or experiment tool.

Strengths

  • Identity verification at sign-up and optional in-study checks for bots and AI-written answers.
  • Hundreds of free prescreening filters and representative samples at no extra cost.
  • Enforces a minimum hourly reward, which simplifies fair-pay requirements in ethics review.

Watch out for

  • An opt-in, non-probability pool, so population estimates need care.
  • Some data-quality guarantees apply only to particular integrations.

Sources:Prolific PricingWho Is on ProlificProlific Data Integrity

CloudResearch Connect

Self-serve participant platform aimed at academic researchers, with US participants and support for multi-wave studies.

Pricing model
Participant pay plus a percentage service fee, lower for academic and non-profit accounts, with an introductory no-fee period for new academic accounts.
Where it runs
Recruitment only; participants are currently in the U.S.

Strengths

  • Participants are vetted at sign-up and monitored on every study for fraud and AI-generated content.
  • Demographic, quota, and scale-score targeting.
  • Longitudinal waves for following the same participants.

Watch out for

  • U.S. participants only.
  • An opt-in, non-probability pool.

Sources:Connect FAQConnect for Researchers

Rep Data Research Defender

Pre-survey fraud screening that scores each respondent before they reach your survey, using device fingerprinting and external reputation checks.

Pricing model
Sold by Rep Data; pricing is not published.
Where it runs
Runs in front of your survey through a redirect or code snippet; documented with Decipher, Qualtrics, and other platforms.

Strengths

  • Device fingerprinting, a list of known fraudsters, and checks for respondents attempting many surveys.
  • A single threat score per respondent that you can act on before the survey starts.

Watch out for

  • A commercial market-research product; validation evidence comes from the vendor.

Sources:Research Defender

Checks inside the survey platform

Qualtrics records a reCAPTCHA bot score and duplicate flags where the license includes Fraud Detection. Domandata records bot scores, duplicate tokens, opt-in VPN and proxy detection, honeypot questions (in beta), paste tracking, and author-defined rules, then routes flags into a review queue with an exportable cleaning log.

Qualtrics Fraud Detection

Qualtrics' built-in fraud tools: a reCAPTCHA-based bot score, post-submission duplicate detection, and cookie-based repeat-submission prevention.

Pricing model
Not included in every license; administrators add it through their account executive.
Where it runs
Cloud, inside Qualtrics.

Strengths

  • Records a bot score, duplicate flags, and private-browsing detection as data fields.
  • Duplicate detection can flag rather than delete, preserving the raw record.

Watch out for

  • Availability depends on your license.
  • The duplicate flag cannot drive branch or display logic during the survey.

Sources:Qualtrics Fraud Detection

DomandataOur product

Browser-based survey platform built for academic social-science research: survey experiments, conjoint, and analysis-ready data.

Pricing model
Free plan for drafting and previewing; fielding a study, Flow logic, and exports need a paid individual or team plan with published self-serve pricing.
Where it runs
Cloud only, on U.S.-based infrastructure providers.

Strengths

  • Randomized treatment arms (weighted, stratified, or balanced) with the assigned condition saved in every export.
  • Conjoint, MaxDiff, list experiments, and reaction-time tasks built in, without custom JavaScript.
  • Native SPSS (.sav) and Stata (.dta) exports with a generated codebook, plus Qualtrics .qsf import.
  • Links OSF preregistrations and deposits replication materials to Harvard Dataverse or Zenodo.

Watch out for

  • Does not sign HIPAA Business Associate Agreements, so it is not suitable for studies that require one.
  • Cannot be self-hosted, and the data-hosting region is not fixed by contract.
  • The free plan cannot collect responses; fielding a study requires a paid plan.
  • A newer, smaller vendor: its SOC 2 Type I audit is still in progress, which some IT reviews ask for.

Sources:Domandata PricingDomandata Trust Center

When fraud risk is unacceptable

Address-based recruitment makes large-scale fabrication far harder than in opt-in pools, at a much higher cost.

Ipsos KnowledgePanel

Probability-based U.S. online panel recruited by address-based sampling, often used for government and academic research.

Pricing model
Commissioned as a custom study; quote-based.
Where it runs
Fielded by Ipsos; households without internet receive a device and access.

Strengths

  • A true probability sample, including households that are offline.
  • Supports population inference that opt-in pools cannot.

Watch out for

  • Commissioned rather than self-serve, and considerably more expensive than opt-in samples.

Sources:KnowledgePanelKnowledgePanel Methodology

Fraud-resistance checklist

  • Exclusion criteria are written into the preregistration
  • Bot score and duplicate signals are recorded as data fields
  • At least two independent attention or consistency checks
  • Minimum time expectations on pages that require reading
  • Paste tracking on open-ended questions that matter
  • A sample of open-ended answers read by a person
  • Flags reviewed, not auto-deleted, with raw data kept
  • A cleaning log exported with the analysis files

Frequently Asked Questions

Do attention checks catch bots?

Not reliably. In Pew Research Center's study of opt-in samples, most bogus respondents passed a standard attention check, and later research shows AI agents can pass attention checks and logic puzzles. Use attention checks as one layer alongside technical signals, consistency checks, and review.

What reCAPTCHA score should I use to exclude responses?

Google describes reCAPTCHA v3 scores from 0.0, very likely a bot, to 1.0, very likely human, and suggests 0.5 as a starting threshold to tune against your own traffic. Record the score, combine it with other signals, and preregister how you will use it rather than excluding on the score alone.

Can AI agents complete online surveys?

Yes. A 2025 study in PNAS showed an autonomous AI agent completing surveys while passing attention checks and common bot-detection tests and keeping a consistent persona. This is why recruitment-side vetting, behavioral signals, and human review matter more than any single in-survey test.

Should I delete flagged responses automatically?

Usually not. Automatic deletion removes the evidence you need to justify exclusions and can drop legitimate respondents, including people using accessibility tools. Flag, review against preregistered rules, keep the raw data, and report how many responses were excluded and why.

References

Keep Reading

Survey Data Quality Checks

Plan quality checks before launch and carry them into exports.

Choosing a Recruitment Platform

Compare vetting and fraud controls across sample sources.

AI in Survey Research

What synthetic respondents and AI interviews can and cannot do.

What Social-Science Surveys Need

Requirements for social-science designs, with pilot tests.