Research Guide

Survey Experiments: Design, Randomization, and Analysis

A credible survey experiment connects the estimand, treatment materials, assignment procedure, sample plan, and analysis before fielding begins. This guide puts those decisions in order and links each one to a practical Domandata walkthrough.

A practical workflow

1
Define the contrast
Write the treatment comparison and primary outcome before choosing a question type or building branches.
2
Build conditions and assignment
Create explicit arms, choose simple, weighted, balanced, or stratified assignment, and save the assigned arm in the export.
3
Plan sample flow
Account for screen-outs, quotas, attrition, and unequal allocation when translating a power target into recruitment counts.
4
Pilot every path
Test treatment exposure, manipulation checks, endings, panel redirects, and the variables needed for the preregistered model.
5
Audit and analyze
Export assignment and display-order fields with the codebook so the realized design can be checked before estimation.

Key design decisions

Decision	Guidance
Independent arms or repeated tasks?	Use condition arms for between-subject treatments; use conjoint, MaxDiff, or repeated vignettes when each respondent evaluates several randomized profiles.
Simple or balanced assignment?	Simple randomization preserves unpredictability. Balanced or stratified assignment is useful when small cells or key covariates could otherwise drift.
When should quotas enter?	Set quotas after the target sample and expected screen-out rate are known; do not use quota counts as a substitute for an assignment audit.