DATA2002 Chap.9 Permutation tests and the bootstrap
Permutation tests and the bootstrap
Every test so far has needed a named distribution for its statistic under the null hypothesis, and that requirement is what forced the assumptions. The previous chapter avoided it by changing the data into ranks. This chapter avoids it differently: keep the data, keep any statistic you like, and construct the null distribution by rearranging what you already hold. The permutation argument is short.
If the group labels carry no information about the values, then any reallocation of those labels to those values was equally likely to have happened, so computing the statistic for every rearrangement produces the distribution it would have under the null, and the p-value is the proportion at least as extreme as what was observed.
The single assumption is exchangeability, the requirement that permuting the observations leaves their joint distribution unchanged. It is weaker than normality and it is not nothing, since data with a time trend or a clustered structure do not satisfy it. Two practical points follow.
The statistic is entirely free, which is the method's main advantage: a difference in means, a difference in medians, a t statistic or a robust ratio all work equally well, and using a t statistic here does not mean using a t distribution. And when the samples are small the rearrangements can be enumerated completely, making the p-value exact rather than simulated. The bootstrap answers a different question.
Where there is no usable formula for the standard error of an estimate, treat the sample as a stand in for the population, resample from it with replacement many times, and read an interval off the quantiles of the resulting spread. It cannot repair a badly collected sample, because every resample comes from that same sample.
What this chapter covers
- 01
The null hypothesis restated as a statement about labels rather than about parameters
- 02
Exchangeability, and the kinds of data that fail it
- 03
Why the statistic is free, and why a t statistic does not bring the t distribution with it
- 04
Enumerating every split when the samples are small, so the p-value is exact
- 05
Sampling the rearrangements at random when enumeration is impractical
- 06
The paired version, which permutes signs rather than labels
- 07
Two candidate paired statistics, and what each one weights
- 08
A robust statistic for data contaminated by outliers
- 09
The bootstrap idea: resample the sample, with replacement, at the same size
- 10
Why the resample must be the same size as the sample
- 11
The percentile interval, and the centred variant for a skewed bootstrap distribution
- 12
Two readings of a confidence interval the unit rules out explicitly
Choose a resampling procedure and say what it estimates
- +2Use the bootstrap percentile method: draw many resamples of size sixty from the sixty observations, with replacement, compute the ninetieth percentile of each, and take the 2.5th and 97.5th percentiles of those values as the interval endpoints.
- +1It will supply an uncertainty estimate for a statistic with no convenient formula, without assuming a distributional shape for the population.
- +1It will not repair a sample that was collected unrepresentatively, because every resample is drawn from that same sample. The bootstrap estimates sampling variability, not bias in how the sample arose.
- +1For reproducibility, report the number of resamples, the random seed, and which interval method was used, since the percentile and centred variants give different endpoints on a skewed bootstrap distribution.
Key terms
- Permutation test
- A test whose null distribution is generated by rearranging the observed data under the null hypothesis, rather than taken from a named distribution.
- Exchangeability
- The requirement that the joint distribution of the observations is unchanged by permuting them. If it fails, the rearrangements are not equally likely and counting them is not a valid probability calculation.
- Exact enumeration
- Computing the statistic for every possible rearrangement rather than a random sample of them, which is feasible when the samples are small and makes the p-value exact.
- Simulation error
- The additional uncertainty in a p-value obtained from a random sample of rearrangements rather than from all of them. It shrinks as the number of rearrangements grows and should be reported alongside that number.
- Sign permutation
- The paired version of the argument, in which each difference was equally likely to have come out negative, so the rearrangements are the sign patterns rather than the label allocations.
- Bootstrap
- Resampling from the observed sample with replacement, at the same size, to estimate the sampling variability of a statistic.
- Percentile interval
- A bootstrap interval whose endpoints are the lower and upper quantiles of the bootstrap distribution corresponding to the chosen confidence level.
- Resample
- One draw of the same size as the original sample, taken with replacement, in which some observations appear more than once and others not at all.
- Robust statistic
- A summary that is not moved far by a small number of extreme observations, such as a difference in medians scaled by a robust measure of spread.
- Random seed
- A value fixing the sequence of pseudo random numbers so that a simulation or a bootstrap can be reproduced exactly.
Permutation tests and the bootstrap FAQ
What is the difference between a permutation test and a bootstrap?
They answer different questions and are used at different points. A permutation test answers a testing question by rearranging labels under a null hypothesis, and it produces a p-value. A bootstrap answers an estimation question by resampling with replacement, and it produces an interval.
The mechanics look similar because both involve repeated resampling, but a question asking whether there is a difference wants the first and a question asking how uncertain a number is wants the second.
If a permutation test uses a t statistic, is it just a t test?
No, and the unit states this explicitly because the combination looks contradictory. What is borrowed is the formula for measuring the effect, not the reference distribution. The permutation test compares that statistic with the distribution generated by rearranging the actual data, so the normality assumption behind the t distribution is never invoked.
In practice the two often agree closely when the t test's own assumptions hold, and the permutation p-value is the more defensible one because it did not need them.
When is a permutation p-value exact rather than approximate?
When every rearrangement is enumerated. With five observations in one group and six in the other there are 462 ways to choose which values form the first group, which is a short list, so the p-value is a complete count rather than an estimate. As the samples grow the number of rearrangements becomes astronomically large and a random sample of them is used instead, which introduces a simulation error and nothing else.
The logic is identical either way.
Does the bootstrap fix small samples?
It makes an interval available where no formula exists, and it does not manufacture information. An interval from twelve observations will be wide, and a bootstrap interval that looks reassuringly narrow on a tiny sample is a warning rather than a result. The method is also silent about how the sample was collected: it resamples what you have, so bias in the original sampling passes through untouched.
How many resamples or rearrangements are enough?
Enough that the answer stops moving at the precision you intend to report. A few hundred is usually adequate for a standard error and a few thousand for the tails of an interval, because a percentile is estimated from the sparse end of the distribution where there are fewest values. The cost is only computer time, so the sensible default is more than you think you need, with the number you actually used reported.
Exam move
Start from the argument rather than from the recipe, because the argument regenerates the recipe for any design you are handed. Three sentences hold it: under the null the labels carry no information, so every reallocation was equally likely, so the collection of statistics across reallocations is the null distribution.
If you can say that, you can rebuild the method even for the paired design, where the thing permuted is the sign rather than the label. Two short things follow from the argument and both are asked directly. Exchangeability, which a question wants named together with one situation that violates it rather than as a list.
And the separation between the two methods of this chapter, which blur under time pressure because their machinery is similar, so hold them apart by the question each one answers. Rehearse the two forbidden readings of a confidence interval alongside the correct one, since they recur across the whole unit and this is the chapter that states them formally.
In your own project work, whenever you meet a statistic with no obvious standard error, treat it as practice, and record the seed and the number of resamples as a habit rather than as an afterthought.
Working through Permutation tests and the bootstrap in DATA2002? Sia is AskSia’s AI Statistics tutor — ask any DATA2002 Permutation tests and the bootstrap question and get a clear, step-by-step explanation grounded in how DATA2002 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
