Open-ended survey response coding: codebook, pilot and audit trail

Open-ended survey response coding converts free text into documented categories that can be reviewed and summarized. Define the unit of coding, preserve the verbatim response, build category definitions with inclusion and exclusion rules, pilot the codebook on a sampled set, double-code enough cases to expose ambiguity, reconcile disagreements, freeze a version and retain a row-level audit trail.

What is open-ended response coding?

Open-ended response coding applies a structured codebook to text units so patterns can be analyzed without discarding the original language. The approach may be inductive, with themes developed from the responses; deductive, with categories specified in advance; or hybrid. It differs from cognitive interviewing, which investigates how respondents understand a question during instrument development.

Define the unit of coding first

Record whether multiple codes may apply, whether repeated mentions count once or many times, and how “don't know,” refusal, off-topic, unintelligible and blank responses differ.

Twelve-field codebook template

1. code_id
2. code_name
3. definition
4. unit_of_coding
5. include_when
6. exclude_when
7. positive_examples
8. counterexamples
9. parent_code
10. multiple_codes_allowed
11. ambiguity_rule
12. version_and_change_note

Keywords can help coders notice relevant text, but they are examples rather than complete definitions. Context determines whether a phrase fits. Store the specification beside the survey data dictionary.

Worked codebook example

Question: “What made checkout difficult?” Response: “The card form failed twice, and I could not find the shipping cost until the final screen.”

Because multiple codes are allowed, the row keeps two binary code fields and two evidence spans while preserving the full verbatim answer.

Eight-step coding protocol

1. Freeze the response export and remove identifiers not required for analysis.
2. Record the eligible open-text universe and item-nonresponse count.
3. Select a development sample using a documented random or purposive rule.
4. Read the sample and draft code definitions with examples and boundaries.
5. Independently double-code a pilot without seeing the other coder's decisions.
6. Compare disagreements, revise definitions and repeat the pilot if material ambiguity remains.
7. Freeze the production codebook, train coders and code the remaining responses.
8. Audit a held-out sample, preserve disagreements and publish the codebook version with limitations.

Agreement and disagreement log

Store response_id, text_version, unit_id, coder_id, codebook_version, assigned_codes, evidence_span, confidence, flag, other_coder_codes, disagreement_type, adjudicated_codes, adjudicator, decision_note and decision_time. Agreement statistics can monitor consistency, but a high number does not prove conceptual validity or eliminate shared misunderstanding.

Worked pilot

Two coders independently apply a five-code book to 100 responses. They agree exactly on 78, disagree on whether 14 responses fit COST_VISIBILITY or NAVIGATION, and disagree on eight other cases. Review shows the COST_VISIBILITY exclusion rule is unclear when users say they “couldn't find” a fee. The team revises both definitions, adds counterexamples and recodes all 100 pilot responses rather than silently carrying old decisions forward.

How should AI-assisted coding be evaluated?

Treat a model, keyword rule or embedding cluster as a coding instrument that needs a versioned prompt or configuration, a frozen codebook, held-out evaluation and human review of disagreements. Preserve the source text and model output separately. Do not claim human-equivalent validity from a few plausible examples, and do not send sensitive text to an unapproved service. Re-run a sample when the model, prompt or codebook changes.

Reusable reporting table

Transparent method

Pew Research Center's 15 September 2026 methodology documents open coding, testing an initial codebook on a random sample of 149 responses, reviewing disagreements, assigning each of the remaining responses to two researchers and permitting multiple applicable codes. Its 2024 methodology shows a larger iterative reliability process across languages, while its 2021 published codebook demonstrates unit-of-information rules and coder instructions. The templates, audit fields, worked pilot and AI boundary here are editorial synthesis derived from those transparent practices, not a universal standard.

Limitations

Codes simplify language and can strip context. Coders can share the same blind spot, translations can change nuance, and rare themes may disappear in a development sample. Item nonresponse can make the open-text subset selective, so coded counts are not automatically population estimates. Illustrative quotations are not a random sample. Sensitive free text may contain identifiers even when structured survey fields do not. Pair coding with an immutable input and row-level issue log from the data-cleaning workflow, and assess subset inference separately with the nonresponse-bias guide.