ARCH9090 Chap.7 Questionnaire Design That Produces Evidence
Questionnaire Design That Produces Evidence
Questionnaire design converts an information need into consistent, analysable responses. This chapter compares closed and open questions, repairs leading and double-barrelled wording, tests response options, anticipates order effects and connects sampling to the claim boundary. It also treats piloting and analysis planning as part of design so apparently polished questions do not produce unusable or misleading evidence.
What this chapter covers
- 01
Linking every questionnaire item to a decision and planned analysis
- 02
Closed questions for comparison and open questions for unanticipated explanation
- 03
Leading, double-barrelled, absolute, technical and ambiguous wording problems
- 04
Balanced, complete and understandable response options with suitable non-applicability
- 05
Question order, priming, respondent burden, device usability and accessibility
- 06
Recruitment and sampling limits on prevalence or population claims
Worked example · free
Repair a leading cycle-lane question
- 1Remove the assumed safety and convenience claims and explain the open decision neutrally. Marks shown in this example are AskSia practice values and are not a University assessment scheme.
- 1Ask current travel exposure before opinions so responses can be interpreted in context.
- 1Separate cycling safety, shop access and parking availability into distinct balanced items.
- 1Add an open reason prompt and keep the effects separate in analysis.
Key terms
- Closed question
- A question offering predefined responses that support consistent comparison.
- Open question
- A question allowing respondents to explain experience or reasoning in their own words.
- Double-barrelled item
- A question that asks about more than one issue while permitting one answer.
- Order effect
- A change in later responses caused by the sequence or framing of earlier questions.
- Pilot test
- A small trial used to identify interpretation, routing, burden, access and analysis problems before launch.
Questionnaire Design That Produces Evidence FAQ
How many questions should a questionnaire contain?
Use only the questions needed to answer the defined information need. Length depends on complexity, device, audience and context. Every item should have a planned analytical use; repeated grids and vague comment boxes create burden without necessarily improving evidence.
When is an open question useful?
Use one when reasons, language or unexpected issues matter and the team has a transparent plan to analyse the text. An open box is not automatically inclusive. It can add burden and produce material that is ignored if coding has not been planned.
Can questionnaire results represent the wider public?
Only when sampling and response support that inference. An open online or convenience questionnaire describes the reached respondents, not an entire population. Report recruitment, missing groups and response patterns, then calibrate chart titles and findings to that evidence boundary.
What should a questionnaire pilot test?
Test interpretation, option fit, order, routing, burden, device usability, accessibility and the exported data structure. Ask pilot participants to explain what they think each question means rather than only whether it seems clear. The team should also rehearse analysis with the trial output.
A question can read smoothly yet fail if its categories do not support the planned comparison or if respondents are forced into inaccurate answers. Recheck routing after revisions, because repairing one item can unexpectedly change the sequence, context or effort shaping later answers.
Assessment move
Build a question ledger with one row per item and columns for information need, question type, response format, planned comparison and likely bias. Remove any item with no analytical use. Practise spotting common flaws by reading drafts aloud: two nouns joined by and often signal a double barrel; approving adjectives often signal leading language; undefined time periods create ambiguity.
Test every response set against a participant who falls outside the expected categories. Pilot the complete sequence on a phone and with someone unfamiliar with the topic, asking them to explain what each item means. Preview the exported data before launch so routing and multiple selections behave as intended. During reporting, label findings as patterns among respondents unless the sampling design supports a broader claim.
Add a response-option audit for completeness, overlap, ordering and non-applicability. Imagine three people whose circumstances do not fit the expected pattern and attempt the survey as each of them. Record where they would be forced into a false answer. Test order by moving a concern prompt before and after a general rating and explain the possible priming effect.
Decide how open text will be coded, reviewed and reported before including it. If the team cannot use the answer, remove or sharpen the prompt. Check that consent and purpose language appear before collection and that the platform does not reveal unnecessary information. After launch, preserve the final instrument and a data dictionary so later recoding is transparent.
When producing charts, write the accurate title first, including the respondent boundary, then select a visual form. This prevents a convenient sample from silently becoming a population claim.