BIOS6902 Chap.9 Potential Outcomes, Randomisation and A/B Tests
Potential Outcomes, Randomisation and A/B Tests
Why eight lectures of estimation were not enough
Frozen-dessert sales and shark bites climb together because both climb when it is hot. Children with larger feet read better because both improve with age. Painkiller users report more headaches because a headache is why the packet was opened.
Each pattern is genuine, correctly measured, with a perfectly valid interval attached, and each would give a disastrous answer to the question of what an intervention would do. Every tool built so far estimates a feature of a distribution; a causal question asks about a different distribution, the one that would hold if a decision were made differently.
The potential-outcomes framework makes the target precise.
For every unit imagine two outcomes: the one that would occur under treatment and the one that would occur under control. The observed outcome is whichever of the two corresponds to the assignment actually received, so exactly one of them is visible for each unit and the individual treatment effect is never observed for anyone.
That is not a limitation of measurement technology; it is a structural feature of the problem, which is why the target shifts to an average.
The average treatment effect, and the quantity it is not
The average treatment effect is the expected difference between the two potential outcomes across units, equivalently the difference of their expectations.
The contrast between the observed outcomes of the treated and the untreated is a different quantity: it compares the people who happened to receive treatment with the people who happened not to. The two coincide only under special conditions, and creating those conditions is what the last three lectures are about.
A chance allocation creates them by design rather than by argument.
Let a coin decide who is treated and allocation becomes independent of both potential outcomes, so knowing which arm a unit landed in reveals nothing about the pair it was carrying. The observable contrast then coincides exactly with the average effect, and what was a comparison of groups becomes a statement about an intervention.
The estimator is the gap between the two arm averages, which for a yes-or-no outcome is a gap between proportions, and the allocation makes it centred on the effect.
Inference, and the discipline that has to come first
Inference can be done with the machinery already built. A difference of independent proportions has a standard error from the two group variances, and an interval follows.
Alternatively, under the sharp null that the treatment changes no unit's outcome, the outcomes are fixed numbers and only the labels are random, so re-randomising the labels many times produces the exact reference distribution of the previous chapter. No normal approximation is required and the argument holds at any sample size, which makes it the natural test for a small experiment.
Design has to precede all of it.
Power rises with the size of the true effect, falls with the variability of the outcome, and rises with the number of units per arm; the lecture's rule of thumb makes the trade explicit and carries a warning inside it, because the target effect appears squared in the denominator. Halving the effect you want to detect multiplies the required sample by four.
Most underpowered experiments are underpowered because someone specified an optimistic effect, not because someone miscounted participants. The five-point checklist that closes the lecture is about exactly this: decide the metric and the rule before the data arrive.
What this chapter covers
- 01
Three classic associations that are not causal
- 02
Potential outcomes and the outcome that is never observed
- 03
The average treatment effect against the observed contrast
- 04
How randomisation makes assignment independent of potential outcomes
- 05
The difference-in-means estimator and its standard error
- 06
Sharp-null re-randomisation as an exact test
- 07
Power, sample size and the pre-launch checklist
Worked example · free
A binary experiment from design to the sentence you report
- 2Write the estimator and compute it from the two observed proportions.
- 2Compute the standard error of a difference of independent proportions.
- 2Assemble the interval.
- 2State the conclusion in terms a decision maker can act on.
Key terms
- Potential Outcome
- The outcome a unit would show under a specified assignment. Each unit has one per arm and only the one matching its actual assignment is observed.
- Average Treatment Effect
- The expected difference between the two potential outcomes across units, which is the usual causal target because individual effects are unobservable.
- Randomisation
- Assigning treatment by a chance mechanism, which makes assignment independent of the potential outcomes and turns an observed contrast into a causal one.
- Difference in Means
- The contrast between the average outcomes of the two arms, unbiased for the causal effect when assignment was randomised.
- Sharp Null
- The hypothesis that treatment changes no unit's outcome at all, which fixes every outcome and leaves only the labels random.
- Primary Metric
- The single outcome nominated before launch as the basis for the decision, protecting the analysis from selection among many measured quantities.
Potential Outcomes, Randomisation and A/B Tests FAQ
Why can both potential outcomes never be observed for one unit?
Because each unit receives exactly one assignment and the observed outcome is the one attached to it. The other is a counterfactual, a description of a world that did not occur, and no measurement can recover it. That is why the individual effect is not an estimable quantity and why the target is an average, where the two unobservable halves can be estimated from different groups of people.
Does a randomised experiment need any assumptions at all?
Fewer than an observational study, but not none. The randomisation must actually have happened as described, units must not interfere with one another in ways that change each other's outcomes, and the measured outcome must be the one the decision depends on. Those are design assumptions rather than distributional ones, which is why they are checked by reading the protocol rather than by fitting a model.
What should a team conclude from a positive estimate with an interval containing zero?
That the experiment has not yet decided the question. The estimate is the best single guess and the interval says which values remain plausible, and when zero is among them the data do not support acting as though an effect exists. The productive response is to say how much more data would be needed for the effect size that matters, which is a design calculation rather than a further analysis.
Exam move
Practise writing the two-sentence conclusion for an experiment: one sentence with the estimate and its interval, one sentence saying what the design does and does not license. The arithmetic in this chapter is easy and the marks are almost entirely in the second sentence.
Working through Potential Outcomes, Randomisation and A/B Tests in BIOS6902? Sia is AskSia’s AI Statistics tutor — ask any BIOS6902 Potential Outcomes, Randomisation and A/B Tests question and get a clear, step-by-step explanation grounded in how BIOS6902 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
