Ranking Survey Questions

Ranking questions ask respondents to order items by preference, importance, priority, likelihood, or perceived value. They are useful in social science research surveys when relative ordering matters more than a single rating for each item.

Use ranking for policy priorities, product features, issue salience, candidate attributes, organizational values, task preferences, or any study where researchers need ordinal preference data.

• Forced tradeoffs reveal relative priorities when rating everything highly would be uninformative.
• Random starting order helps reduce primacy effects in repeated ranking tasks.
• Recode values keep items stable in exports even if labels are revised before launch.
• Clear item lists make ranking easier for participants and easier to interpret later.
• Instructions: an optional box shown above the question, for directions such as "Drag the items so your top priority is first." It is translatable like the question text.
• Up/down move buttons sit next to the drag handle for every respondent, not only ones on touch devices — a tap-only alternative to dragging, since WCAG 2.2's Dragging Movements criterion treats a keyboard shortcut alone as insufficient for a touchscreen user who has no keyboard.

1. Step 1: Open Block Builder. Open the survey and select the block where the preference task belongs.

   Step 1: Open Block Builder

2. Step 2: Add a Ranking question. Choose Ranking from the question type selector.

   Step 2: Choose Ranking question type

3. Step 3: Add the items. List the items respondents should order, and keep each label short enough to scan on mobile.

   Step 3: Add ranking items

4. Step 4: Set starting order. Use question options to randomize, reverse, or fix the starting order based on the design.

   Step 4: Set starting order

5. Step 5: Show a subset instead of the full list, if the item pool is long. Turn on Show subset and choose how many items each respondent ranks; Domandata draws a fresh random subset per respondent instead of showing every item to everyone, which keeps each individual respondent's task short even when the full item pool is long. Use the lock and anchor icons next to any item to always include it in every respondent's subset (lock) or always show it in the same position (anchor) — useful for a "none of the above" or brand-you-always-want-ranked item.

   Step 5: Show a subset and lock or anchor items

6. Step 6: Allow ties, if a strict order would misrepresent the respondent's actual view. Turn on Allow ties and a tie icon appears next to each item (after the first) that a respondent can use to mark it as equally important as the item above it; tied items share the same displayed rank number. Research on tied ranks (Timbrook & Moroney, Survey Practice 2016) found most respondents used a tie option when offered one and preferred it over a strictly forced order.

   Step 6: Allow ties

7. Step 7: Allow partial ranking, if respondents shouldn't have to place every item. Turn on Allow partial ranking and respondents see two lists — "Your Ranking" and "Not Ranked" — and move only the items they care to rank into the first one. Set a minimum and/or maximum number of items a required question will accept; Instrument Review warns if the minimum is higher than the maximum. Without partial ranking, a required question asks the respondent to either reorder the items or choose Keep This Order, so an untouched list is never submitted by accident. Stored answers are unaffected by this setting's absence: they're the same ordered list of ranked items either way, just possibly shorter than the full item count.

   Step 7: Allow partial ranking

8. Step 8: Add recodes if needed. Use stable item names or recodes when the ranking will feed analysis across waves or treatments. Row shading (off or alternating) is a display-only option for scanability and has no effect on exported data.

   Step 8: Add recodes

9. Step 9: Preview the task. Open Preview and make sure the item count is not too burdensome, especially on mobile.

   Step 9: Preview the ranking task

Ranking answers export as an ordered list. For analysis, decide in advance whether lower rank numbers mean higher priority and document that choice in your codebook. If ties are on and a respondent actually marked one, that row's answer is instead an object with a parallel list of which items are tied; unranked items from a partial ranking are simply absent from the list, not stored as null. If you need independent ratings instead of forced ordering, consider Slider survey questions or Grid Matrix survey questions. Note that ranking (and forced-choice data generally) is ipsative — each respondent's values are interdependent by construction — which can distort ordinary factor analysis by introducing artificial negative correlations between items (Davidov & Meuleman, International Journal of Public Opinion Research 2019); a ratings battery has the opposite bias (uniformly high ratings), so neither format is analysis-neutral.

Response quality automatically flags a couple of ranking-specific patterns, with no setup needed: a response left in the exact order the items were displayed in (no reordering at all), and a ranking completed implausibly fast for its item count. Both are heuristic signals meant to surface for review, not proof of inattention on their own — a respondent can legitimately agree with the displayed order. Instrument Review separately nudges when an item list runs long enough (past roughly 5–7 items) that reliability in the middle of the ranking may suffer; that threshold is a practitioner rule of thumb, not a validated cutoff. Consider a random item subset (see Step 5 above) or MaxDiff for a longer item pool.

• Conjoint Survey Questions for preference experiments
• Constant Sum Survey Questions when the magnitude of a tradeoff matters
• MaxDiff Survey Questions for priorities across many items
• Slider Survey Questions
• Create Experimental Conditions
• Publish and Share a Research Survey