ECMT1010 · Introduction To Economic Statistics
Sampling, Bias & Study Design
Week 1 sets the logic the whole unit runs on: we use a sample to make an inference about a population, and the two things that can break that inference are sampling bias (a non-random selection method) and confounding (a lurking third variable). It is examined as MCQ in the Week-7 test and as a short-answer 'classify the study and justify a causal claim' question, where the marks live in the two-question decision tree — random selection licenses generalisation, random assignment licenses causation.
What this chapter covers
- 011. Population vs sample, and inference; parameter (Greek) vs statistic (Latin)
- 022. Sampling bias and why random sampling fixes it (the Dewey/Truman polling failure)
- 033. Other biases: measurement, response and survivorship bias
- 044. Association vs causation — and why association alone never proves cause
- 055. Confounding variables: the lurking third variable that blocks a causal claim
- 066. Observational study vs experimental study (random assignment + control)
- 077. Randomization, blinding (single/double/triple) and the placebo effect
- 088. Natural experiments (the NJ minimum-wage / Card study) and the two-question decision tree
Classify the studies and justify a causal claim
- 2 marks(a) Study 1 is observational — members chose for themselves whether to use the app, so the explanatory variable was not assigned. Study 2 is an experimental study (a randomized controlled trial) because app use was randomly assigned.
- 3 marks(b) Study 1: no causal claim is justified — without random assignment a confounder could drive both app use and more exercise. Study 2: a causal claim is justified — randomization balances confounders across the two groups on average, so the difference in workouts can be attributed to the app.
- 1 mark(c) A plausible confounder is baseline motivation/fitness: members who choose to use the app may already be keener exercisers, inflating the apparent effect.
Key terms
- Population vs sample
- The population is every individual or object of interest; the sample is the subset actually measured. Inference is the act of using the sample to draw a conclusion about the population.
- Sampling bias
- When the selection method makes the sample systematically differ from the population on something relevant, so inferences are wrong. Random sampling is the cure — it makes the sample representative on average.
- Association vs causation
- Two variables are associated if knowing one tells you something about the other; they are causally associated if changing one actually changes the other. Association does not imply causation.
- Confounding variable
- A third variable associated with both the explanatory and the response variable. It offers an alternative explanation for an observed association and is the reason an observational study cannot establish cause.
- Observational vs experimental study
- In an observational study we record variables as they naturally occur; in an experimental study the researcher randomly assigns the explanatory variable. Only random assignment removes confounders and licenses a causal conclusion.
- Blinding
- Hiding group membership to remove bias. Single-blind hides it from subjects (removes the placebo effect); double-blind also hides it from researchers (reduces observer bias); triple-blind also hides it from the analyst (reduces confirmation bias).
Sampling, Bias & Study Design FAQ
Why does random sampling matter so much?
Because the whole point of the unit is to generalise from a sample to a population, and that only works if the sample is representative. A non-random selection method introduces sampling bias — the classic example is a poll that systematically over-samples one type of voter and so calls the election wrong. Random sampling gives every member of the population a chance of selection, so on average the sample mirrors the population.
When can I claim one thing causes another?
Only when the explanatory variable was randomly assigned, i.e. in a randomized experiment. Random assignment balances all confounders (known and unknown) across the groups on average, so a difference in the response can be attributed to the treatment. In an observational study a confounder can always offer a rival explanation, so you may only claim association.
What is the difference between sampling bias and a confounder?
Sampling bias is about who gets into the sample — a flawed selection method that makes the sample unrepresentative, which threatens generalisation. A confounder is a lurking third variable linked to both the explanatory and response variables, which threatens a causal claim. They map onto the two separate questions in the decision tree: random selection fixes the first, random assignment fixes the second.
What is a natural experiment?
A natural experiment is an observational situation where an external event assigns a treatment almost as if it were random — for example a policy change applied in one place but not a comparable neighbour, as in the New Jersey minimum-wage study. It can support stronger causal claims than an ordinary observational study, but because the assignment was not deliberately randomized you must still argue carefully that the groups were comparable.
Exam move
Make the two-question decision tree automatic, because almost every Week-1 mark hangs off it: random selection → generalisation, random assignment → causation. Drill the vocabulary so you can label a described study instantly as observational or experimental and name the explanatory and response variables, then practise generating a specific confounder for any observational claim (a concrete lurking variable earns the mark; 'other factors' does not). Learn the bias family — sampling, measurement, response, survivorship — with one example each, and be ready to explain in a sentence why blinding and a placebo control remove particular biases. For the conceptual MCQs, the trap is the seductive causal headline drawn from observational data; train yourself to ask 'was anything randomly assigned?' before agreeing.
