Estimate Sample Size With Deploy Power

Beta: Deploy Power is currently available to beta testers while its design compiler, simulations, and planning workflow are refined.

The Power card in Deploy separates two questions that are often confused: how many observations the statistical comparison needs, and how many total respondents must enter the survey for every analysis cell to reach that number. It reads supported outcome questions, conjoint structure, and Flow assignments from the authored survey, proposes an analysis, then applies the saved assumptions and allocation paths.

Choose Find Min N to work backward to a total recruitment target. Choose Set N to enter a planned total and inspect achieved power and the minimum detectable effect (MDE). Select the actual terminal Flow cells used by the primary comparison; if none are selected, Domandata plans conservatively against the smallest terminal cell.

Open How This Was Calculated beneath the results for a live, survey-specific audit trail. It lists the structure Domandata detected, the method and effective alpha it used, the method-specific equation or design effect, the sequential attrition and exclusion adjustment, and how it solved backward through the authored Flow allocation. The explanation updates whenever the survey design or an assumption changes and states when a result is only an approximation.

Compiled Design turns the survey itself into a methodological checklist before it calculates power. It identifies the selected outcome and scale, the share of entrants who can reach it, randomized question-subset inclusion, respondent-level assignment, repeated tasks or loop rounds, the plausible hypothesis family, estimated survey burden, and risks from optional questions, quotas, response-quality rules, and time limits. When panel fields or prior-wave answers are linked, it also identifies them as possible pre-treatment blocking variables or covariates.

Each item has two labels. Survey means it was read directly from the instrument, Derived means Domandata calculated it from several authored settings, Plan means it is a researcher-entered assumption, and Pilot means it came from a saved pilot summary. Exact describes the authored design, Estimated depends on future respondent behavior, and Confirm marks an analytic choice the survey cannot decide.

Outcome exposure combines authored Flow reach with random question-subset inclusion. It deliberately excludes future item nonresponse and treats respondent-dependent Skip Rules as uncertain. Domandata shows this exposure as an audit warning rather than silently multiplying it into the recruitment target: a skip rule may define the analysis population, post-treatment missingness, or both, and those cases require different assumptions. Use pilot completion and missingness rates to set attrition or exclusions explicitly.

• Independent Means compares continuous outcomes in independent groups with a finite-sample noncentral t calculation and can account for unequal allocation and variance explained by preregistered covariates.
• Binary Proportions compares two independent probabilities. Enter the baseline probability and a signed difference defined as alternative minus baseline; Domandata rejects an alternative outside zero to one instead of silently truncating it.
• Paired / Repeated Means uses the expected within-person correlation and a finite-sample paired t calculation.
• Clustered Means applies a design effect using the intraclass correlation (ICC), average cluster size, and cluster-size coefficient of variation, then searches over whole clusters using cluster-level degrees of freedom.
• Conjoint AMCE appears automatically when the instrument contains a clickable forced-choice conjoint. Enter the smallest average marginal component effect, expressed as a probability difference from 0 to 1.

Domandata lists supported outcome candidates and records the chosen question in the plan. It recognizes clickable conjoint choices, numeric sliders, reaction-time outcomes, two-category choices, and repeated matrix responses. It warns instead of guessing when a MaxDiff or compositional constant-sum estimand needs a specialized analysis.

Target power and alpha are separate inputs. You can choose a one- or two-sided hypothesis, Bonferroni or conservative first-threshold Holm familywise error control, and whether power means one comparison, at least one discovery, or all confirmatory comparisons. The family power objectives use an explicit independence approximation because correlations among future test statistics cannot be inferred from the questionnaire alone.

For a paired forced-choice conjoint, Domandata reads the authored question rather than asking you to re-enter its structure. It detects the two profiles, the number of repeated tasks, the attribute with the most levels, and any fixed holdout tasks. Holdouts are excluded because they validate a fitted model but do not add randomized profile assignments for estimating an AMCE.

The conjoint calculation follows Schuessler and Freitag's analytic power equation for forced-choice AMCEs. Its effective sample size is:

respondents × randomized tasks × substantive profiles per task

Domandata calculates the required effective profile evaluations and divides by the randomized tasks and profiles to return the required respondents per analysis cell. If the survey has several conjoint questions, choose the design that defines the confirmatory estimand; required N, achieved power, simulation, cost optimization, and exports all use that same selected design.

The analytic calculation is methodologically appropriate for a standard forced-choice AMCE with randomized profiles. The seeded Design Simulation generates method-specific binomial counts or continuous sufficient statistics and reports power, a separately simulated false-positive rate, confidence-interval coverage, bias, RMSE, Type S error, Type M exaggeration, and Monte Carlo uncertainty. If Domandata detects an opt-out, unequal level probabilities, or prohibited combinations, it labels the result as an approximation: the final preregistration should additionally simulate the exact choice model and fitted estimator.

1. Step 1: Finalize the main comparison. Specify the estimator, smallest effect of interest, alpha, power, sidedness, and multiplicity plan.
2. Step 2: Select the comparison cells. The card stores stable Flow-cell identifiers and applies the authored allocation to those paths instead of silently treating every branch as a study arm.
3. Step 3: Add expected losses. Attrition and post-recruitment exclusions are applied sequentially, so the displayed recruitment target is larger than the analyzable target.
4. Step 4: Check sensitivity. Review the power curve and conservative, expected, and optimistic effect/attrition scenarios. A single point estimate is rarely enough.
5. Step 5: Run the seeded simulation. Increase the run count until the displayed Monte Carlo uncertainty is acceptably small, then retain the seed.
6. Step 6: Save and export. Save Analysis Plan stores the assumptions with the survey. Export its versioned JSON, R and Python scaffolds, and preregistration text for review.

Expected allocation is not automatically treated as guaranteed. Even balancing uses its deterministic count. Ordinary random assignment uses the binomial allocation distribution, and Find Min N chooses a total that gives every selected comparison cell at least a 95% marginal allocation guarantee, splitting the failure allowance across the selected cells. Deterministic or respondent-dependent rules that cannot be known before collection are labeled as estimated and receive the same conservative allocation treatment.

Import a CSV from a pilot study and choose a numeric outcome column. Domandata calculates the complete-case sample size, mean, sample standard deviation, and—when the column contains only zeroes and ones—the baseline probability. Raw pilot rows remain in the browser; only the selected summary is saved in the analysis plan.

For a conjoint, enter the fixed cost per respondent and the incremental cost per task. The respondent–task optimizer holds the AMCE target and profile count fixed, evaluates alternative task counts, and ranks them by estimated cost. The cheapest row is not automatically the best design: task fatigue, carryover, and response quality can make a shorter battery preferable.

• Schuessler, Julian, and Markus Freitag. 2020. Power Analysis for Conjoint Experiments. Source for the forced-choice AMCE power equation and effective profile sample.
• Hainmueller, Jens, Daniel J. Hopkins, and Teppei Yamamoto. 2014. Causal Inference in Conjoint Analysis. Source for the AMCE estimand and profile-level analysis framework.
• Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences. General reference for conventional two-group power analysis.
• Eldridge, Sally M., Deborah Ashby, and Sally Kerry. 2006. Sample size for cluster randomized trials: effect of coefficient of variation of cluster size and analysis method. Source for the unequal-cluster-size design-effect adjustment.
• Gelman, Andrew, and John Carlin. 2014. Beyond power calculations: assessing Type S and Type M errors. Source for sign and magnitude diagnostics.
• Lakens, Daniël. 2022. Sample Size Justification. Guidance on smallest effects of interest, sensitivity analysis, and transparent justification.

An analytic calculator and method-specific simulation are planning aids, not evidence that every planned analysis is powered. The unit, outcome scale, allocation ratio, estimator, clustering, exclusions, and multiplicity correction must match the confirmatory analysis. The exported R and Python files reproduce the selected built-in method, analyzable cell size, degrees of freedom, and conjoint exposure assumptions; adapt them when the preregistered estimator is more complex than the selected method.

• Conjoint Analysis Survey Design
• Sample Allocation Across Condition Arms
• Create Experimental Conditions
• Set Survey Quotas