Evidence asset
Survey question quality checklist: 25 tests before launch
A reproducible 25-point test for detecting leading, vague, double-barrelled and hard-to-answer survey questions before fieldwork.
- Published
- 24 August 2026
- Reading time
- 12 min
- Author and reviewer
- The Survey Review
A survey question is launch-ready only when it is necessary, neutral, understandable, answerable and analyzable. Score every question against the 25 tests below. Revise any item that fails a critical test, even if the total score looks acceptable.
What is survey question quality?
Survey question quality is the degree to which a question captures the intended concept without avoidable wording, recall, response-option or context error. A polished interface cannot rescue a question that measures the wrong thing.
The 25-point question test
| # | Test | Pass condition |
|---|---|---|
| 1 | Purpose | The question maps to one named decision or analysis field. |
| 2 | Necessity | Removing it would create a real evidence gap. |
| 3 | One concept | It does not combine two ideas with and or or. |
| 4 | Neutrality | It does not signal a preferred answer. |
| 5 | Plain wording | A respondent can understand it on first reading. |
| 6 | Defined terms | Specialist or ambiguous terms are explained. |
| 7 | Specific period | Behavior questions name a recall window. |
| 8 | Answerable | The target respondent can reasonably know the answer. |
| 9 | No assumption | It does not presume an event or opinion. |
| 10 | No absolute | It avoids unrealistic words such as always or never. |
| 11 | Balanced scale | Positive and negative positions receive comparable space. |
| 12 | Distinct options | Response choices do not overlap. |
| 13 | Complete options | Common valid answers have a place. |
| 14 | Opt out | Not applicable or prefer not to answer appears when needed. |
| 15 | Ordered labels | Every scale point has a clear label and logical order. |
| 16 | Consistent direction | High and low values mean the same thing across related items. |
| 17 | No hidden ranking | A select-all item is not treated as a priority ranking. |
| 18 | Limited burden | The requested detail matches what respondents can recall. |
| 19 | Mobile length | The stem and options remain usable on a narrow screen. |
| 20 | Translation ready | Idioms and wordplay are avoided. |
| 21 | No context leak | Earlier questions do not reveal the desired response. |
| 22 | Safe sensitivity | Sensitive questions are justified and placed deliberately. |
| 23 | Logic fit | Every answer works with the branch it can trigger. |
| 24 | Analysis fit | The response format supports the intended calculation. |
| 25 | Pretested | At least one person outside the authoring team interpreted it as intended. |
How to score a questionnaire
- Score each question independently with 1 for pass and 0 for fail.
- Mark critical failures separately.
- Calculate the average score across all questions, but do not average away a critical failure.
- Ask a person who did not write the survey to explain each question in their own words.
- Record revisions and rerun the test before the pilot.
Worked example: fix a double-barrelled question
Draft: “How satisfied are you with our delivery speed and packaging?” This fails the one-concept test because a respondent may like the speed but dislike the packaging.
Revision: Ask two items: “How satisfied are you with the delivery speed?” and “How satisfied are you with the condition of the packaging?” Keep the same fully labelled response scale so the results can be compared.
What this checklist cannot prove
A high score does not establish validity, eliminate sampling error or show that respondents interpret a specialized construct consistently. New or important measures still need cognitive interviewing, pilot testing or other pretesting appropriate to the stakes.
Sources and limitations
- Pew Research Center: Writing Survey Questions, including wording, response options, order effects and pretesting.
- UK Government Analysis Function: survey design quality, including guidance on leading and double-barrelled questions.
- US Census Bureau: Comparing Pretesting Methods, on cognitive interviews, respondent debriefing and behavior coding.
Verification date: 24 August 2026. This is operational survey-design guidance, not legal advice. Requirements can differ by jurisdiction, audience and research purpose. Send corrections with a primary source through our corrections process.