SOAD2021 Chap.6 Quantitative Design and Analysis
Quantitative Design and Analysis
Quantitative design translates concepts into variables that can be measured across defined units. Begin with a precise question, population, outcome, comparison and time frame. Operational definitions determine what each variable represents; reliability and validity determine whether repeated measurement is consistent and whether the intended concept is captured.
Sampling affects inference.
Probability approaches support population estimates when the sampling frame and response process are credible. Non-probability approaches may be feasible for service evaluation but require narrower claims. Surveys must minimise ambiguous wording, double questions, leading frames and response options that fail to cover the experiences being measured.
Analysis begins with distributions, not a significance test.
Inspect frequencies, centre, spread, outliers and missingness. A sample mean is x bar equals the sum of observed values divided by n. A proportion is p hat equals the number with the characteristic divided by the number observed. Inferential procedures then assess uncertainty under assumptions.
A p-value does not measure effect size, importance or the probability that a hypothesis is true.
Design controls the question a calculation can answer. A cross-sectional survey can describe an association at one period but cannot establish which condition came first. Repeated observations can show change, yet change may reflect attrition, measurement shifts or events outside the program.
A comparison group can strengthen reasoning only when group formation and concurrent conditions are understood. Before analysis, draw the pathway from target population to sampling frame, invited sample, respondents and analysed cases; annotate loss at each step. After analysis, report the estimate in meaningful units, an uncertainty measure where appropriate, and the consequences of missingness and design.
Practical importance must be considered with participants and decision makers rather than inferred from a threshold. When testing a hypothesis, pre-state the contrast and assumptions, then treat the result as one component of a broader evidentiary judgement instead of a binary declaration.
What this chapter covers
- 01
Operational definition
- 02
Variables
- 03
Sampling
- 04
Survey design
- 05
Descriptive statistics
- 06
Hypothesis testing
- 07
Statistical interpretation
Worked example · free
Worked example: separate size, precision and importance
- 1State the observed mean difference: the new pathway group waited 3 fewer days in the sample.
- 1Interpret p = 0.04 as compatibility under the specified null model, not the probability that the pathway works.
- 1Request interval estimates, sample size, distribution, missingness and design details to assess precision and bias.
- 1Ask whether a 3-day difference is meaningful for clients and service decisions.
- 1Avoid causal language unless allocation and design address plausible alternative explanations.
Key terms
- Operational definition
- An operational definition specifies how a concept is represented, measured or classified in a study. It enables scrutiny of whether the variable fits the intended construct.
- Sampling frame
- A sampling frame is the practical list or process from which a sample is selected. Gaps in the frame limit who can be represented by the resulting evidence.
- P-value
- A p-value is the probability, under a specified model and null hypothesis, of a result at least as incompatible with that model as the observed result. It is not an effect size.
Quantitative Design and Analysis FAQ
What is the difference between a sample and population?
A population is the full set of units to which a question refers, while a sample is the observed subset. Generalisation depends on how the sample was reached, who did not participate, measurement quality and whether the target population matches the sampling frame.
Should I report the mean for every numerical variable?
No. Inspect the distribution first. Strong skew, outliers or bounded scales can make the mean misleading without additional information. Report a measure of centre with an appropriate measure of spread and enough context for the reader to understand the units and sample.
What does statistical significance tell me?
It describes data compatibility with a specified null model under assumptions and a chosen procedure. It does not establish causation, practical importance, measurement validity or replicability. Interpret the estimate, uncertainty, design and decision context together rather than treating a threshold as a verdict.
Can a convenience sample still be useful?
Yes, for feasibility, description of the participating group or generation of questions. Its limitation is the inference, not automatic worthlessness. Describe recruitment, compare available characteristics with the intended population and avoid presenting participant estimates as population estimates without justification.
Assessment move
For each quantitative question, write the unit, population, exposure or group, outcome, time and intended comparison. Create a small invented dataset and calculate a count, proportion and mean by hand before using software. Sketch the distribution and mark missing values. Practise interpreting a result in four sentences: estimate, uncertainty, practical meaning and design limit.
Finally, critique one survey item for ambiguity, assumptions and response coverage, then rewrite it. Change the sampling process in the scenario and explain exactly which population claim must narrow even when the computed estimate stays identical. Recalculate after removing one extreme value, then discuss the sensitivity without deleting the observation from the real analysis.
