MAST20034 · Critical Thinking With Data
Sampling and WEIRD Bias
Week 9 asks the question behind every survey result: does this sample honestly represent the population? A sample is a guess about a population, and the method decides whether the guess is honest. The chapter nails the core vocabulary — unit, population, sampling frame, sample — and the gaps between them (the frame can miss part of the population, the sample can miss part of the frame). The box model is the engine: random sampling is like drawing tickets from a box, and that is what makes the sample's behaviour predictable. The mark-earning distinction is sampling error vs bias: sampling error is random and shrinks with size; bias is systematic and a bigger sample only entrenches it. You learn the sampling-method taxonomy — the random methods (simple random, stratified, cluster, systematic) and the non-random ones (convenience, purposive, quota, snowball) — and the biases that ambush non-random sampling: non-response, undercoverage, voluntary response. The chapter closes on WEIRD samples (Western, Educated, Industrialised, Rich, Democratic) and the generalisability crisis, training the reflex you apply to any sample: who's missing?
What this chapter covers
- 019.1 The core vocabulary — unit, population, sampling frame, sample
- 029.2 The box model — what random sampling actually does
- 039.3 Sampling error vs bias — the distinction that earns marks
- 049.4 The sampling-method taxonomy (random vs non-random methods)
- 059.5 WEIRD samples, non-response and the ‘who's missing?’ reflex
Naming the sampling method and its bias, mark by mark
- +1Name the method: recruiting whoever volunteers from an easily reached group is convenience sampling (a non-random method), here also voluntary-response.
- +1Name the biases: the frame is undergraduates at one university — undercoverage of the wider population — and only the willing reply, adding voluntary-response/self-selection bias.
- +1Name WEIRD: the sample is the textbook WEIRD profile (Western, Educated, Industrialised, Rich, Democratic), so it cannot stand in for ‘human decision-making’ in general.
- +1Why size won't help + fix: these are biases, not random error, so more students only entrench them; generalising needs a probability sample of the target population, or explicit limits on the claim.
Key terms
- Population / frame / sample
- The population is everyone you want to conclude about; the sampling frame is the list you actually draw from; the sample is who you get. Gaps between them — frame missing population, sample missing frame — are where coverage bias lives.
- Box model
- The teaching device for random sampling: imagine drawing tickets from a box. It makes the sample's variability predictable and underpins the idea of sampling error as the spread of repeated draws.
- Sampling error vs bias
- Sampling error is the random difference between a sample estimate and the truth — it shrinks as n grows. Bias is a systematic tilt that a larger sample cannot cure. Confusing them is the classic ‘big sample = accurate’ mistake.
- Probability vs non-probability sampling
- Probability methods (simple random, stratified, cluster, systematic) give every unit a known non-zero chance of selection, enabling honest inference. Non-probability methods (convenience, quota, purposive, snowball) do not, and invite selection bias.
- WEIRD sample
- A sample drawn from Western, Educated, Industrialised, Rich, Democratic populations — common in research yet a poor stand-in for humanity at large. The canonical external-validity / generalisability problem.
Sampling and WEIRD Bias FAQ
Why can't a big sample fix sampling bias?
Because bias is systematic — it shifts the estimate's centre away from the truth — while sample size only reduces random scatter around that centre. A huge biased sample is a precisely wrong answer; only an unbiased design fixes it.
What's the fastest way to critique a sample in the exam?
Ask ‘who's missing?’. It immediately surfaces undercoverage, non-response and WEIRD limitations, after which you name the sampling method and separate bias from random error.
What is a WEIRD sample and why does it matter?
One drawn from Western, Educated, Industrialised, Rich, Democratic groups. It matters because findings from such samples often don't generalise to other populations, so claims about ‘people’ in general are over-reaches.
Exam move
Put the sampling-method taxonomy on your notes sheet as two columns (random vs non-random) with each method's one-line risk, and keep the sampling-error-vs-bias one-liner right beside it. Drill the ‘who's missing?’ reflex on every sample prompt — it is the fastest route to undercoverage, non-response and WEIRD. Be able to define population/frame/sample and point to the gap a given scenario exploits. Rehearse the ‘bias, not error, so a bigger sample won't help’ argument verbatim; it answers a recurring exam trap.
