DATA2002 Chap.10 Multiple testing and error rate control
Multiple testing and error rate control
Nothing in this chapter says any individual test was wrong. Each one, at the five per cent level, rejects a true null hypothesis five per cent of the time, exactly as designed. The problem is that a study rarely runs one test, and the guarantee that holds for each test separately says nothing about the collection.
Ten thousand tests on data where no effect is present anywhere produce, in expectation, five hundred discoveries, all of them false, and that is not a pathological example but the definition of the significance level multiplied by the number of tests. Even twenty independent tests give about a 64% chance of at least one false alarm.
The chapter sets out an outcome table for many hypotheses, of which only two quantities are ever observable, the number of tests and the number declared significant, and then defines three error rates on it. The false positive rate is the per test guarantee you already have.
The family wise error rate is the probability of at least one false positive anywhere, and the false discovery rate is the expected proportion of the claims actually made that are false. Those last two are very different in practice: a list of a hundred discoveries with five false ones violates the family wise criterion and has a perfectly acceptable false discovery rate. Two procedures follow, one attached to each.
The Bonferroni correction divides the level by the number of tests, or equivalently multiplies each p-value by it, and bounds the family wise rate at the cost of being conservative, meaning its true error rate sits below the advertised one. The step up procedure compares each ordered p-value with a threshold that rises down the list and controls the false discovery rate instead.
What this chapter covers
- 01
The outcome table for many tests, and which of its six quantities you can ever see
- 02
The arithmetic that starts the argument: level times number of tests
- 03
The false positive rate, the family wise error rate and the false discovery rate, side by side
- 04
How fast the family wise rate grows with the number of independent tests
- 05
Which procedure controls which rate, stated as the pairing to memorise
- 06
The Bonferroni threshold, and the equivalent adjusted p-value
- 07
Conservative, defined as a true error rate below the advertised level
- 08
Choosing the family the correction applies to, and why that is a judgement
- 09
Ordering p-values and comparing each with a rising threshold
- 10
Why the step up rule takes the largest passing rank and everything above it
- 11
Where each procedure is at home, in terms of what happens to the list next
- 12
The caveat about strong dependence between tests
Compute a family wise rate and a corrected threshold
- +1Each test individually fails to raise a false alarm with probability 0.95, and the tests are independent, so none of the eight raises one with probability 0.95 to the power eight.
- +1.5The probability of at least one is therefore one minus 0.95 to the power eight, which is about 0.337. Roughly a one in three chance of a spurious finding on data where nothing is happening.
- +1.5The Bonferroni corrected threshold is 0.05 divided by 8, which is 0.00625. Equivalently, multiply each p-value by eight and compare with 0.05.
- +1The corrected family wise rate is one minus 0.99375 to the power eight, which is just under 0.05 rather than exactly 0.05. That gap is what makes the method conservative, and the room it leaves is power that has been given up.
Key terms
- Family
- The collection of tests a correction is applied across. Choosing it is a judgement rather than arithmetic, and a correction is only as meaningful as the family it is stated for.
- Family wise error rate
- The probability of at least one false positive anywhere in a family of tests. It grows quickly with the number of tests and is what the Bonferroni correction bounds.
- False discovery rate
- The expected proportion of the results declared significant that are false. Controlling it is more permissive than controlling the family wise rate.
- False positive rate
- The rate at which genuinely null results are called significant, which is the per test guarantee the significance level already provides.
- Bonferroni correction
- Dividing the significance level by the number of tests, or equivalently multiplying each p-value by it. It bounds the family wise error rate using the union bound.
- Adjusted p-value
- A p-value rescaled so that it can be compared with the original significance level rather than with a corrected threshold. It is the same decision expressed differently.
- Conservative
- Describing a procedure whose true Type I error rate sits below the advertised level. The guarantee is met with room to spare, and the room is paid for in power.
- Step up procedure
- A method that orders the p-values, compares each with a threshold proportional to its rank, and declares the largest passing rank together with everything above it.
- Discovery
- A test declared significant. The number of discoveries is observable; the number of them that are false is not, which is why every error rate is a property of a procedure.
- Dependence between tests
- Correlation among the test statistics in a family. Strong dependence can cause problems for both corrections, and it is the caveat the unit closes the topic on.
Multiple testing and error rate control FAQ
Which procedure controls which rate?
Bonferroni controls the family wise error rate, and the step up procedure controls the false discovery rate. The pairing is easy to swap under pressure, which is exactly why it is examinable. Attach each name to its own sentence: Bonferroni to at least one false positive, and the step up rule to the proportion of my discoveries that are false. If you can say those two sentences, both procedures follow from them.
How do I decide how many tests to correct for?
By deciding what the family is, and then saying so. Dividing by the three comparisons you planned is a different act from dividing by the four hundred you could have run, and neither is automatically right. State the family and why, because a reader cannot check the correction without it. The related failure is choosing the family after seeing which tests came out well, which is a way of tuning the correction to the result.
When would I prefer to control the false discovery rate?
When the output is a shortlist that will be followed up anyway. A screen returning a hundred candidates of which five are wrong is a good day, and controlling the family wise rate on a study of that size would leave you with almost nothing.
Control the family wise rate instead when a single false claim is expensive, which is the situation with a small number of pre specified comparisons in a study that will be acted on directly.
Does the step up rule just mean keeping every p-value under its threshold?
No, and this is the part most often got wrong. The rule finds the largest rank whose p-value sits under its own threshold and then declares that one together with everything smaller. If a later rank passes while an earlier one does not, the later one still pulls the earlier one in. Testing each rank independently and keeping only those that pass is a different procedure and does not control the rate it claims to.
Are these corrections valid when the tests are related to each other?
Both are stated for independent tests, and strong dependence between tests can cause problems, which the unit flags as the closing caveat on the topic. For this unit a basic correction of one of the two kinds is the expected answer, and noticing the dependence issue is worth a sentence in a longer response rather than a change of method. Multiple testing under dependence is an entire subfield.
Exam move
Two things here are worth memorising and two are worth practising. The number to memorise is how fast the family wise rate climbs: at twenty independent tests it is about sixty four per cent, which is worth being able to quote because it makes the whole argument concrete in a sentence.
The pairing to memorise is which procedure controls which rate, learnt in both directions, so that a question naming the procedure and a question naming the rate are equally answerable.
What to practise is one example worked fully in both procedures on the same list of p-values, because seeing that the strict method returns two results and the tolerant one returns five from identical evidence is what makes the distinction stick, and then stating in plain language what each procedure controls, since a question asking you to explain the choice to a non specialist is asking for exactly that and not for the arithmetic.
One sentence belongs in every answer you write in this chapter: the family the correction was applied to, named explicitly, because that sentence is often worth a mark on its own.
Working through Multiple testing and error rate control in DATA2002? Sia is AskSia’s AI Statistics tutor — ask any DATA2002 Multiple testing and error rate control question and get a clear, step-by-step explanation grounded in how DATA2002 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
