BIOS6902 Chap.8 P-values, Model Fit and Permutation Inference
P-values, Model Fit and Permutation Inference
Why a p-value replaces a pre-committed threshold
Fixing a significance level in advance forces a binary decision, and different choices can send identical data in opposite directions. The lecture's own illustration is a statistic of one and seven tenths, which fails a five per cent threshold and clears a ten per cent one.
Reporting the p-value removes the need to pre-commit and tells the reader how close the call was. The definition is the smallest level at which the null would be rejected, equivalently the probability under the null of a statistic at least as extreme as the one observed.
Everything about the definition is conditional on the null being true, which is the source of the four standard misreadings.
A large p-value does not establish that the null is true; a small study of a real effect produces large p-values routinely. The number is not the probability that the null is true, because reversing the conditioning requires a prior and takes you back into the Bayesian chapter. Statistical significance is not importance.
And the gap between forty-nine and fifty-one thousandths is not a boundary in nature, which is why the lecture advises reporting p-values as numbers and avoiding rigid significant-or-not language.
Power, and the single quantity that controls it
The chapter also prices detection.
Under regularity the estimate is asymptotically normal with a variance set by the information, so a Wald statistic scaled that way rejects beyond a normal critical value. If the truth differs from the null, a single shift term determines everything: it grows with the square root of the sample size, with the information each observation carries, and with the distance between the truth and the null.
Near the null that shift is close to zero and the power sits barely above the significance level, which is not a defect but a fact about small effects. As the sample grows without bound the power tends to one, which is also why very large studies detect effects too small to act on.
Goodness of fit asks a different question: did these data come from a fully specified distribution?
Reduce the sample to counts in categories, or bin a continuous variable, and compare observed with expected counts through Pearson's statistic, which under the null is approximately chi-square with one fewer degree of freedom than there are categories. The single degree lost is the one spent forcing the expected counts to total the sample size.
Check that no expected count is very small before trusting the approximation.
Building the reference distribution instead of assuming it
Comparing two groups admits a modelled route and a model-free one. The modelled route puts both groups inside a family and asks whether their parameters agree, using either a scaled distance or a ratio of maximised likelihoods.
The relabelling route assumes only that, if the groups really are alike, any assignment of the labels was as likely as any other.
Work out the statistic once, then work it out again under many reshuffles of the labels, and count what share of the reshuffles reach or pass the value you started with.
With three observations the whole reference set fits on one line: six reshuffles, four of which reach or pass the value you started with, giving an exact p-value of four sixths.
Nothing was approximated anywhere, because the reference distribution is the complete list rather than a limiting curve. At realistic sample sizes the list is too long to enumerate, so a large random sample of rearrangements is used and the p-value becomes arbitrarily accurate rather than exact.
The price is computation; the return is Type I error control without a distributional assumption, which is why this method reappears immediately in the causal chapters.
What this chapter covers
- 01
The p-value as the smallest level at which the null would be rejected
- 02
One-sided and two-sided tail probabilities
- 03
The power of a Wald test and the shift term that controls it
- 04
Four standard misreadings of a p-value, with their corrections
- 05
Pearson's chi-square statistic and its degrees of freedom
- 06
Two-sample Wald and likelihood-ratio tests
- 07
Permutation inference and the exact reference distribution
Worked example · free
A goodness-of-fit test taken through to its interpretation
- 2Multiply each reference proportion by the sample size to get expected counts.
- 1Check that no expected count is too small for the approximation.
- 3Form each squared discrepancy over its expectation and sum them.
- 2State the degrees of freedom and interpret the comparison.
Key terms
- P-value
- The smallest significance level at which the null would be rejected, equivalently the null probability of a statistic at least as extreme as the observed one.
- Two-sided Test
- A test whose rejection region covers both tails, appropriate when departures in either direction are of interest.
- Goodness of Fit
- An assessment of whether observed data are consistent with a fully specified distribution, usually through category counts.
- Pearson Statistic
- The sum over categories of squared differences between observed and expected counts, each divided by its expected count.
- Expected Count
- The sample size multiplied by a category's hypothesised probability, which must not be very small for the chi-square approximation to hold.
- Permutation Test
- A test that builds its reference distribution by recomputing the statistic under rearrangements of the group labels, assuming only exchangeability.
- Exchangeability
- The assumption that, under the null, every arrangement of the labels is equally likely, which is what licenses a permutation reference distribution.
P-values, Model Fit and Permutation Inference FAQ
Does a large p-value mean the null hypothesis is true?
No. It means the data are compatible with the null, which is also what a small study of a genuine effect produces. The correct sentence is that the evidence against the null is insufficient at this level. Report the estimate and its interval alongside the p-value so a reader can see which values were ruled out and which remain plausible.
When should relabelling replace a modelled comparison?
When there are few observations, the outcome is visibly lopsided, or the statistic has no convenient distribution to compare against. All it needs is that the labels could equally have fallen the other way if the groups were alike, which a chance allocation guarantees by construction; the modelled route leans on an approximation that a small lopsided sample may not support.
How many rearrangements should I use?
Enough that the p-value is stable when you rerun with a different seed. Small p-values need more rearrangements than large ones, because they are estimated from the sparse extreme of the reference set. Report the number used; a permutation p-value without it cannot be judged.
Why does Pearson's statistic lose one degree of freedom?
Because the expected counts are constrained to total the sample size, so once all but one category is known the last is determined. Only the number of categories minus one can vary freely, and the approximating distribution is indexed by that count.
Exam move
Write the p-value definition on your notes sheet word for word. Every procedure in this chapter is a different way of computing that one probability, from a normal approximation, from a chi-square approximation, or by counting rearrangements, so a precise definition tells you which computation a question wants.
Working through P-values, Model Fit and Permutation Inference in BIOS6902? Sia is AskSia’s AI Statistics tutor — ask any BIOS6902 P-values, Model Fit and Permutation Inference question and get a clear, step-by-step explanation grounded in how BIOS6902 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
