MaxDiff Survey Questions

MaxDiff (best-worst scaling) shows respondents a small subset of items at a time and asks which is best and which is worst. Repeated across several trials, it produces sharper priority data than a single Likert battery because respondents are forced to discriminate between similarly rated items instead of rating everything highly. It is well suited to prioritization studies, feature or message testing, and issue-importance research with many candidate items.

Use MaxDiff when you have too many items for a clean ranking task and need to know which items separate from the pack. If you only have a handful of items, Ranking or Grid Matrix may be simpler for respondents.

• Item pool: the full list of items being evaluated (use at least four).
• Items per trial: how many items are shown together in each set.
• Number of trials: how many sets each respondent completes.
• Best / worst labels: customize the column headers respondents see (for example, "Most important" and "Least important").
• Anchor items: guarantee a specific item appears in every trial (for example, a reference item you want compared against everything else) — a real, respondent-facing every-trial-inclusion guarantee, not the same thing as “Anchored MaxDiff” (see below).
• Instructions: an optional box shown above the question, for task directions such as "Pick the most and least important feature in each set." It is translatable like the question text.
• Attention-check set: an optional instructed-response set — see below.
• Balanced-random trial design: Domandata draws each respondent's trial sets from a shuffled pool so item exposure stays roughly even across the sample, without requiring a hand-built experimental design.

1. Step 1: Open Block Builder. Open the survey and select the block where the prioritization task belongs.

   Step 1: Open Block Builder

2. Step 2: Add a MaxDiff question. Choose MaxDiff from the question type selector.

   Step 2: Choose MaxDiff question type

3. Step 3: List the item pool. Add every item under consideration. A larger pool benefits more from MaxDiff than a short list does.

   Step 3: List the item pool

4. Step 4: Set items per trial and trial count. Four to five items per trial and four to six trials is a common starting point; more trials improve precision but add respondent burden.

   Step 4: Set trial design

5. Step 5: Name and code it. Add a readable variable name before piloting so the per-trial columns are easy to interpret in export.

   Step 5: Variable Name

6. Step 6: Preview the trials. Confirm each set shows a sensible number of items and that best and worst selections are mutually exclusive within a trial.

   Step 6: Preview the trials

Turn on Add an attention-check set to make one set an instructed-response check. Choose which set, the instructed item, and whether respondents must mark it Best or Worst, and optionally add instructions shown above that set. The chosen set always includes the instructed item. A respondent who picks anything else is flagged with the Failed Attention/Manipulation Check quality flag. Leave that set out when you estimate item scores or utilities, since its answer is instructed rather than a real preference.

The options panel shows an item coverage estimate: given your item pool, items per trial, and number of trials, it reports how many times each item appears across one respondent's own trials. Each item's comparative standing can only be judged from the trials it actually appeared in, so a large item pool relative to items-per-trial × trials under-covers some items — the more central design lever for MaxDiff than trial count alone. As a rough guide, roughly 3 appearances per item per respondent is the point below which individual-level (per-respondent) scoring generally isn't feasible later, even though Domandata doesn't compute that scoring itself.

Enter an assumed respondent count to also see the total appearances per item across your expected sample. Published guidance for how many total appearances an aggregate analysis needs is genuinely split across sources — roughly 200 on the conservative end, 500 or more for higher precision — so treat the number as a planning range, not a single confident target. Instrument Review separately flags a design whose per-respondent appearance rate falls below the individual-level floor, regardless of whether you've entered a respondent count.

Each response exports the item shown in every trial along with which one the respondent marked best and which they marked worst. Domandata does not automatically compute best-minus-worst scores or a utility model — plan to aggregate best/worst counts per item during analysis, the standard approach for MaxDiff data.

Response quality automatically flags a few MaxDiff-specific patterns, with no setup needed: a malformed trial (a best or worst pick that isn't one of the shown items, or the same item marked both), a respondent who repeatedly picks the same displayed position as best or worst across trials (items reshuffle every trial, so this is a genuine attention signal, not a coincidence), and a battery completed implausibly fast given the item and trial count. These are heuristic signals, not a substitute for a full consistency-based (e.g. hierarchical Bayes) respondent-quality check if your analysis needs that level of rigor.

• Constant Sum Survey Questions for magnitude-based tradeoffs
• Ranking Survey Questions
• Survey Variable Names and Recodes
• Export Research Survey Responses