Survey nonresponse bias assessment: diagnostics and sensitivity checklist

Nonresponse bias occurs when survey respondents and nonrespondents differ in a way that changes a target estimate. A response rate measures participation; it does not quantify bias by itself. Assess the risk by naming the estimate, comparing respondents with frame or administrative variables, examining response timing and contact effort, and testing how defensible assumptions about missing cases would change the result.

What is survey nonresponse bias?

For a named estimate, nonresponse bias is the systematic difference between the expected respondent-based estimate and the target-population value caused by response patterns. Its direction and size can differ by outcome in the same survey. A statement such as “the survey has 10% nonresponse bias” is incomplete without the estimate, reference value and assessment method.

Start with a risk statement

Record the target population, sampling frame, field period, response-rate definition, target outcome, variables known for the full sample and plausible reasons response might relate to that outcome. Link the disposition counts to the response-rate worksheet, but keep the participation measure separate from the bias assessment.

Diagnostic evidence ladder

Worked frame-comparison example

A frame contains 2,000 cases: 1,200 in group A and 800 in group B. The survey receives 600 A responses and 240 B responses. Group response rates are 50% and 30%, and respondents are 71.4% A even though the frame is 60% A. If satisfaction differs by group, the unadjusted overall estimate is exposed to composition bias. This identifies a mechanism to investigate; it does not reveal how nonrespondents within either group would answer.

Worked sensitivity calculation

Suppose 840 respondents report 70% positive, with 1,160 nonrespondents. If nonrespondents would be 50% positive, the full-sample hypothetical proportion is (840 × .70 + 1,160 × .50) ÷ 2,000 = 58.4%. At a 60% assumption it becomes 64.2%. These are not corrected estimates; they show how a conclusion depends on declared assumptions.

Reusable assessment worksheet

Capture estimand, target_population, frame_version, disposition_definition, response_rate, full_sample_covariates, benchmark_source, diagnostic_method, observed_difference, adjustment_model, sensitivity_scenarios, decision_impact, limitations, analyst and verification_date.

Eight release checks

1. State the exact response-rate formula and dispositions.
2. Name the outcome or estimate whose bias is being assessed.
3. Verify comparison variables exist for respondents and nonrespondents.
4. Align benchmark definitions, universe and reference date.
5. Report missingness in every diagnostic variable.
6. Compare weighted and unweighted estimates with the weight specification.
7. Run at least two defensible missing-case scenarios when the decision is sensitive.
8. Preserve code, frozen inputs and a table of assumptions.

Transparent method

AAPOR's task-force report says response rate alone cannot establish whether nonresponse bias is present and recommends multiple approaches: comparisons with external data, internal variation such as contact attempts, and alternative adjusted estimates or sensitivity assumptions. It also treats bias as estimate-specific. The evidence ladder, worksheet and arithmetic scenarios here are editorial synthesis of those principles, not universal thresholds.

Limitations

Diagnostics can miss bias on unobserved variables. Administrative benchmarks can contain their own error. Follow-up samples can remain selective. Calibration can reduce one discrepancy and leave another. Opt-in samples introduce selection issues beyond conventional unit nonresponse. Complex designs require design-consistent variance and may need specialist analysis.