Survey weighting guide: design weights, post-stratification and diagnostics

A survey weight tells an analysis how much a responding unit represents. Start with the inverse of the unit's selection probability when a probability design provides one, apply documented nonresponse adjustments, and calibrate only to compatible population controls. Before reporting, inspect extreme weights, effective sample size and how key estimates change with and without weighting.

What is survey weighting?

Survey weighting assigns each analysis row a numeric contribution so estimates reflect a defined target population and documented sample design. In a probability sample, the base or design weight is generally the reciprocal of the final selection probability. Later adjustments may address nonresponse or align weighted totals with external controls.

Weighting is an analysis step, not a substitute for sampling documentation. Link the final weight to the data dictionary, preserve fieldwork dispositions from the response-rate worksheet, and keep the unweighted respondent count beside every weighted estimate.

Survey weight component matrix

How do you calculate a design weight?

For a sampled unit with final selection probability p, the simple base weight is 1 ÷ p. A unit selected with probability 0.02 receives a base weight of 50 and represents 50 units under the design before later adjustments. In a multistage sample, the final probability is the product of the relevant stage probabilities, so use the documented final inclusion probability rather than one convenient stage.

Worked post-stratification example

Suppose a sample contains 600 Group A respondents and 400 Group B respondents, while defensible population controls put each group at 50% of the target population. With 1,000 respondents, the target count is 500 per group. The simple adjustment factors are 500/600 = 0.8333 for A and 500/400 = 1.25 for B. Applying them gives weighted counts of approximately 500 and 500.

The arithmetic fixes this chosen marginal distribution. It does not show that respondents represent nonrespondents within A or B, correct missing population coverage, or validate the control totals. Record the population universe, reference period and category mapping before applying the factors.

Reusable weight specification

Record one specification before calculating the final variable: weight_name, target_estimand, analysis_universe, source_population, base_weight_formula, selection_probability_fields, adjustment_cells_or_model, control_totals_version, calibration_method, convergence_rule, trimming_rule, normalization, missing_control_treatment, software_version, analyst and verification_date.

Seven diagnostics before analysis

1. Confirm each final weight is numeric, finite, positive and attached to the intended analysis row.
2. Recompute a sample of base weights from the documented inclusion probabilities.
3. Compare weighted totals with every population control, using the same universe and categories.
4. Report the minimum, selected percentiles, median, mean, maximum and coefficient of variation.
5. Calculate Kish's unequal-weighting approximation, then use design-based variance procedures when strata or clusters matter.
6. Compare important estimates unweighted, weighted and under defensible trimming alternatives.
7. Reproduce the full weight from frozen inputs, code, software version and control totals.

Worked effective-sample-size check

For weights 0.5, 0.5, 1.0, 1.0, 2.0 and 2.0, the sum of weights is 7 and the sum of squared weights is 10.5. Kish's approximate effective sample size is (Σw)² ÷ Σ(w²) = 49/10.5 = 4.67, below the six observed cases. This compact check describes unequal-weighting variance under simplifying assumptions; it is not a replacement for estimate-specific variance that accounts for clustering and stratification.

Transparent method

We converted the private draft into a reproducible specification and source audit. The CDC/NCHS weighting tutorial documents the sequence of reciprocal-probability base weights, nonresponse adjustments and post-stratification. Statistics Canada covers point estimation, variance estimation and data processing in its survey methods manual. Pew's comparison shows that raking, matching and propensity weighting are distinct approaches whose results depend on the adjustment variables and sample context. The component matrix, arithmetic examples and release checks are editorial synthesis; they are not universal thresholds.

Limitations

Weighting cannot create coverage for people absent from the frame, recover information never collected, remove all nonresponse bias or turn an opt-in sample into a probability sample. Sparse cells, poorly aligned control totals and extreme adjustments can increase error. Replicate weights, imputation, multistage designs, small-area estimates and model-based inference require methods tailored to the design. Do not use Kish's approximation as the only reliability test for a complex survey.