All Research Guides

A Guide From Domandata

AI in Survey Research: What the Evidence Supports

Written by Domandata, which makes a survey platform. We name our product where it fits and say where another tool fits better. Claims about other tools link to their own documentation, checked ; confirm anything decisive with the vendor or your institution.

Survey platforms are adding AI quickly: simulated respondents, AI-led interviews, automated coding of open ends, and question drafting. Some of these uses are well supported by evidence and some are not, and the difference matters for what you can claim in a paper.

This guide summarizes the main research on language models as survey respondents, the American Association for Public Opinion Research's 2026 guidance, and the uses that hold up.

The short answer

Use AI-simulated respondents for clearly labeled pretesting: checking routing, exports, and analysis code before fielding. Do not use them as a substitute for human respondents when you want to estimate what people think; studies find they understate variation, shift with prompt wording and model version, and misrepresent some groups. AI-led interviews show promise for qualitative depth when disclosed and consented. Whatever you use, disclose it, following AAPOR's 2026 checklist.

What the research shows

  1. 1

    Models can mimic group averages

    Argyle and colleagues (2023) showed that a language model conditioned on demographic backstories can reproduce some subgroup response patterns, which they called algorithmic fidelity and the resulting data silicon samples.

  2. 2

    But they are not reliable for inference

    Bisbee and colleagues (2024) found that model-generated responses roughly matched average feeling-thermometer scores but showed less variation, produced regression estimates that often differed from the human data, and changed with prompt wording and over time.

  3. 3

    Whose opinions they reflect is uneven

    Santurkar and colleagues (2023) found substantial misalignment between language models and many U.S. demographic groups, persisting even when the model was steered toward a group.

  4. 4

    Professional guidance is cautious

    AAPOR's 2026 task force report warns that synthetic responses pose serious validity and disclosure risks beyond clearly labeled pretesting, pilot work, or exploratory diagnostics, and that AI interviewers complicate standardization.

  5. 5

    AI interviews can add depth

    Geiecke and Jaravel report that an open-source AI interviewer produced interviews rated highly by respondents and experts across several domains, suggesting a role for qualitative probing at scale with appropriate safeguards.

  6. 6

    AI also threatens human data

    Participants and automated agents can use AI to answer surveys. Protecting real responses is part of using AI responsibly; see the companion guide on bot and fraud detection.

Uses that hold up, and tools that support them

Pretesting logic, exports, and analysis code

Simulated respondents are useful for dry runs: confirming every branch is reachable and that analysis code runs on the export. Domandata's Silicon Samples are designed for this and tag generated rows so they never mix with real data. Qualtrics' Synthetic Panels target attitudinal questions for a single U.S. English population; treat outputs as exploratory and disclose them.

DomandataOur product

Browser-based survey platform built for academic social-science research: survey experiments, conjoint, and analysis-ready data.

Pricing model
Free plan for drafting and previewing; fielding a study, Flow logic, and exports need a paid individual or team plan with published self-serve pricing.
Where it runs
Cloud only, on U.S.-based infrastructure providers.

Strengths

  • Randomized treatment arms (weighted, stratified, or balanced) with the assigned condition saved in every export.
  • Conjoint, MaxDiff, list experiments, and reaction-time tasks built in, without custom JavaScript.
  • Native SPSS (.sav) and Stata (.dta) exports with a generated codebook, plus Qualtrics .qsf import.
  • Links OSF preregistrations and deposits replication materials to Harvard Dataverse or Zenodo.

Watch out for

  • Does not sign HIPAA Business Associate Agreements, so it is not suitable for studies that require one.
  • Cannot be self-hosted, and the data-hosting region is not fixed by contract.
  • The free plan cannot collect responses; fielding a study requires a paid plan.
  • A newer, smaller vendor: its SOC 2 Type I audit is still in progress, which some IT reviews ask for.

Sources:Domandata PricingDomandata Trust Center

Qualtrics Synthetic Panels

AI-generated responses from Qualtrics' own model, offered as a panel distribution for attitudinal and intent questions.

Pricing model
Requires Qualtrics' newer packaging or its preview program, paid with subscription credits.
Where it runs
Cloud, inside Qualtrics; currently U.S. general-population, English only.

Strengths

  • Fast feedback on forward-looking attitude and intent questions before fielding.
  • Qualtrics documents its limits and advises disclosing that results are AI-generated.

Watch out for

  • Many question types are unsupported, and it is weak on past behavior and recall by the vendor's own description.
  • One population and one language at present.

Sources:Qualtrics Synthetic Panels

AI-led qualitative interviews

Domandata's Text Chat question runs a model-led conversation inside a survey with a system prompt you write, turn limits so interviews stay comparable, and paste tracking. Disclose the AI interviewer in consent materials and keep transcripts with the data.

DomandataOur product

Browser-based survey platform built for academic social-science research: survey experiments, conjoint, and analysis-ready data.

Pricing model
Free plan for drafting and previewing; fielding a study, Flow logic, and exports need a paid individual or team plan with published self-serve pricing.
Where it runs
Cloud only, on U.S.-based infrastructure providers.

Strengths

  • Randomized treatment arms (weighted, stratified, or balanced) with the assigned condition saved in every export.
  • Conjoint, MaxDiff, list experiments, and reaction-time tasks built in, without custom JavaScript.
  • Native SPSS (.sav) and Stata (.dta) exports with a generated codebook, plus Qualtrics .qsf import.
  • Links OSF preregistrations and deposits replication materials to Harvard Dataverse or Zenodo.

Watch out for

  • Does not sign HIPAA Business Associate Agreements, so it is not suitable for studies that require one.
  • Cannot be self-hosted, and the data-hosting region is not fixed by contract.
  • The free plan cannot collect responses; fielding a study requires a paid plan.
  • A newer, smaller vendor: its SOC 2 Type I audit is still in progress, which some IT reviews ask for.

Sources:Domandata PricingDomandata Trust Center

Keeping AI out of human responses

Both platforms screen for AI-generated content, and Prolific offers opt-in authenticity checks for bots and AI-written free text. Combine them with paste tracking and review in your own survey.

Prolific

Self-serve online participant pool for academic and commercial research, with participants mostly in OECD countries.

Pricing model
Pay as you go: participant rewards plus a percentage platform fee, lower for academic and non-profit accounts with an institutional email. No monthly fee.
Where it runs
Recruitment only; links to a study built in any survey or experiment tool.

Strengths

  • Identity verification at sign-up and optional in-study checks for bots and AI-written answers.
  • Hundreds of free prescreening filters and representative samples at no extra cost.
  • Enforces a minimum hourly reward, which simplifies fair-pay requirements in ethics review.

Watch out for

  • An opt-in, non-probability pool, so population estimates need care.
  • Some data-quality guarantees apply only to particular integrations.

Sources:Prolific PricingWho Is on ProlificProlific Data Integrity

CloudResearch Connect

Self-serve participant platform aimed at academic researchers, with US participants and support for multi-wave studies.

Pricing model
Participant pay plus a percentage service fee, lower for academic and non-profit accounts, with an introductory no-fee period for new academic accounts.
Where it runs
Recruitment only; participants are currently in the U.S.

Strengths

  • Participants are vetted at sign-up and monitored on every study for fraud and AI-generated content.
  • Demographic, quota, and scale-score targeting.
  • Longitudinal waves for following the same participants.

Watch out for

  • U.S. participants only.
  • An opt-in, non-probability pool.

Sources:Connect FAQConnect for Researchers

Before you use AI in a study

  • The role of AI is stated in the preregistration and methods section
  • Synthetic data is labeled and never pooled with human responses
  • Claims about people rest on human respondents
  • AI interviewers are disclosed in consent materials
  • Prompts, model names, and versions are recorded
  • AI coding of open ends is validated against human coding
  • Disclosure follows the AAPOR checklist

Frequently Asked Questions

Can language models replace survey respondents?

Not for estimating what people think. Research finds model-generated responses understate variation, can produce different regression results from human data, shift with prompt wording and model version, and misrepresent some demographic groups. They are useful for labeled pretesting, not as evidence about a population.

Are silicon samples useful at all?

Yes, for checking a survey before fielding: routing, quotas, exports, and analysis code. AAPOR's 2026 guidance treats clearly labeled pretesting, pilot work, and exploratory diagnostics as the defensible uses. Report any synthetic data separately from human data.

How should I disclose AI use in a survey study?

AAPOR's 2026 task force proposes a disclosure checklist that includes which tasks used AI, how many human respondents took part, and how outputs were validated and overseen. Record model names, versions, and prompts so others can evaluate and reproduce the work.

Is Domandata's Silicon Sample a substitute for real respondents?

No. Domandata's own documentation says Silicon Samples are not a substitute for human pilot testing and should not be used to infer population behavior. They exist to test survey logic, exports, cleaning workflows, and analysis pipelines before real data collection.

References

Keep Reading

Bot and Fraud Detection

Keep bots and AI-written answers out of human data.

Silicon Samples in Domandata

Generate, tag, and remove simulated test responses.

AI Text Chat Questions

Configure a model-led conversation inside a survey.

Survey Software for Open Science

Preregistration, instrument records, and replication packages.