BIOS6902 Chap.11 Trustworthy A/B Testing: Peeking and Multiple Metrics
Trustworthy A/B Testing: Peeking and Multiple Metrics
Where trustworthy experiments actually break
Controlled online experiments run in software products, in digital health and behaviour-change programmes, in pricing and promotion, and increasingly in public-health work. Underneath sits the two-arm randomised comparison of the previous chapter and nothing more.
What changes is the setting: numbers accumulate hour by hour, a dashboard quietly records dozens of outcomes, and somebody wants a verdict long before the planned sample has arrived. The course puts the conclusion bluntly, and it is worth repeating in an answer: what breaks these experiments is almost always the process around them rather than the statistics inside them.
Five habits are singled out.
Inspecting the numbers before the sample is complete. Scanning many outcomes and publishing only the striking ones. Swapping which outcome counts once the results are visible. Halting as soon as the curve looks encouraging. Treating any detectable shift as a reason to act.
Each one either raises the rate of false alarms or tilts the estimate in a flattering direction, and the downstream cost is wasted engineering effort and product decisions taken on random fluctuation; in research the same habits feed the reproducibility crisis.
The arithmetic behind repeated looks
If you test as data arrive and stop at the first result below the threshold, the true false positive rate can be far above the level you thought you were spending.
Each look is another opportunity to cross the line, and the procedure is designed to stop on the first crossing. The independent case makes the accounting visible: the chance of never crossing in a fixed number of looks is the complement of the level raised to that power, so the chance of crossing at least once grows quickly.
Real sequential looks are correlated and the true inflation is smaller, but it is inflation nonetheless and it grows with every additional look.
Screening many outcomes is the same arithmetic in different clothing. Put twenty measures through the usual threshold with no adjustment and a single false alarm is what you should expect on average, even in a world where the change did absolutely nothing.
Twenty inspections of one measure and one inspection of twenty measures are the same ledger entry, and they take the same remedy.
Corrections, and the discipline that makes them work
The plainest adjustment divides the threshold by however many inspections or outcomes were planned.
It gives away power whenever the comparisons are related, which is why designs that lose less exist and why the course names several of them. One principle sits beneath all of them: nothing about how the verdict will be reached may be chosen once the numbers are visible.
An adjustment applied afterwards to a value that was picked out for being small does not restore the guarantee it appears to restore.
The closing checklist is a design document rather than an analysis plan. Nominate one outcome and one threshold for acting, before launch. Work out how many units per arm a shift worth acting on would require. Allocate by a genuine chance mechanism and check the arms came out balanced.
Do not inspect early; where interim inspection is unavoidable, adopt a design that has already paid for it. Publish the estimate, its interval and what it means in practice, not a bare verdict. Write the plan down and keep it where others can read it.
The course also insists that ethics belongs inside the design: a gain measured over a fortnight can hide damage that shows up over a year, participants who are already vulnerable need not respond like the average one, and openness with participants and commissioners is part of running an experiment properly rather than a separate compliance step.
What this chapter covers
- 01
Where online experiments are used, and what the environment adds
- 02
Five named pitfalls and what each does to the error rate
- 03
The arithmetic of repeated looks in the independent case
- 04
Multiple metrics as the same accounting problem
- 05
Dividing the level by the number of planned comparisons
- 06
Sequential designs that pay for interim looks properly
- 07
Pre-registration, reporting discipline and experimental ethics
Worked example · free
Pricing a dashboard of twelve metrics
- 2Compute the chance that at least one of twelve independent tests is significant under the null.
- 2Apply the correction and compare with the observed value.
- 1Say what may legitimately be reported.
- 2State the procedure that would turn the signal into evidence.
Key terms
- Peeking
- Examining results before the planned sample is reached and stopping at the first significant outcome, which inflates the false positive rate.
- Family-wise Error Rate
- The probability of at least one false rejection across a set of related tests, which grows with the number of comparisons unless it is controlled.
- Bonferroni Correction
- Dividing the significance level by the number of planned comparisons, a simple and conservative way to control the family-wise error rate.
- Sequential Design
- A plan that specifies interim analyses in advance with adjusted thresholds, allowing early stopping without breaking error control.
- Pre-registration
- Recording the primary metric, the sample size and the decision rule before launch, so the analysis cannot be selected after the data are seen.
- Practical Significance
- Whether an effect is large enough to change a decision, which is a separate question from whether it is statistically distinguishable from zero.
Trustworthy A/B Testing: Peeking and Multiple Metrics FAQ
Why does an early inspection break the error rate?
Because every inspection is a fresh opportunity to cross the threshold and the rule is built to halt at the first crossing. The guarantee attached to a single test assumes one comparison, and taking several without adjustment spends the error budget many times over. Fixing the decision rule in advance, or using a design that allocates the budget across planned looks, restores the guarantee.
Is dividing the level by the number of tests always the right correction?
It is always valid and often unnecessarily strict, because it assumes the worst case in which the comparisons are unrelated. When tests are correlated, as repeated looks at accumulating data always are, it costs more power than the situation requires, which is why dedicated sequential boundaries and false-discovery approaches exist for the cases where power matters.
What should be reported when a secondary metric moves but the primary one does not?
Report the primary metric as the decision and the secondary movement as an observation requiring confirmation. Describe it as hypothesis-generating, give its uncorrected and corrected results, and list every metric that was examined. Converting it into evidence requires a fresh experiment in which it is the pre-specified primary metric.
How does this chapter connect to the rest of the course?
It closes the arc. Six lectures built estimation, likelihood and prior updating; two built testing and the p-value; the last three put all of it to work on questions about what an intervention would actually do. This chapter supplies the process discipline without which the preceding machinery produces confident, reproducible-looking nonsense.
Exam move
Write out a short pre-registration paragraph for an imaginary experiment: primary metric, expected control rate, smallest lift worth acting on, sample size per arm, analysis, stopping rule. Doing it once makes every question in this chapter a matter of reading your own paragraph back.
Working through Trustworthy A/B Testing: Peeking and Multiple Metrics in BIOS6902? Sia is AskSia’s AI Statistics tutor — ask any BIOS6902 Trustworthy A/B Testing: Peeking and Multiple Metrics question and get a clear, step-by-step explanation grounded in how BIOS6902 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
