DATA2002 Chap.11 One-way ANOVA and post hoc comparisons
One-way ANOVA and post hoc comparisons
With three or more groups the obvious move is a t test on every pair, and the previous chapter explains why that is a poor instrument: the comparisons multiply, the family wise error rate climbs, and each test throws away the information in the groups it is not looking at.
The analysis of variance replaces them with a single question asked once, and it estimates the underlying spread from all the data at the same time, which is a genuine gain: with three groups of six, a standalone pairwise comparison has ten degrees of freedom for its variance estimate while the pooled one has fifteen. The method rests on one identity.
Every observation's distance from the grand mean splits into where its group sits and where it sits inside its group, and squaring and summing makes the cross terms vanish, so the total sum of squares divides exactly into a treatment part and a residual part, with the degrees of freedom dividing alongside.
Each part becomes a mean square by division, and the F ratio asks whether the between group part is large relative to the within group part. That relative comparison is why a raw difference between group means cannot answer the question. Rejecting tells you that the means are not all equal and nothing about which differ, so the second half of the chapter is the follow up.
A contrast is a weighted combination of group means with weights summing to zero, and a pairwise comparison is the simplest kind, using the shared variance estimate from the table. Three multiplier methods then differ less in size than in scope: one is valid for contrasts fixed in advance, one is built for the all pairwise family, and one is exact for every contrast and therefore permits looking at the data first.
What this chapter covers
- 01
Why several t tests is the wrong instrument for more than two groups
- 02
What pooling the variance across all groups buys the comparisons
- 03
The decomposition identity, and why the cross terms vanish
- 04
Sums of squares and degrees of freedom both dividing, with two free arithmetic checks
- 05
The F ratio as a relative rather than an absolute comparison
- 06
The two degrees of freedom in order, numerator first
- 07
Why only the upper tail is used, as in the chi-squared tests
- 08
The contrast statistic, and the pairwise special case
- 09
The union bound, and why the resulting method is conservative by construction
- 10
The quantile argument that carries a factor of two for two sided intervals
- 11
Three multipliers compared by what each one permits rather than by size
- 12
Checking assumptions on the residuals, and four repairs matched to four failures
Complete an analysis of variance table from two numbers
- +1.5There are four groups and twenty eight observations. Treatment degrees of freedom are the number of groups minus one, which is three; residual degrees of freedom are observations minus groups, which is twenty four; total is twenty seven, and three plus twenty four confirms it.
- +1The residual sum of squares is the total minus the treatment part, 312.0 minus 96.0, which is 216.0.
- +1Treatment mean square is 96.0 over 3, which is 32.0. Residual mean square is 216.0 over 24, which is 9.0.
- +0.5The F ratio is 32.0 over 9.0 on three and twenty four degrees of freedom. Note that the sums of squares add and the degrees of freedom add, but the mean squares do not, because each is a different sum divided by a different divisor.
Key terms
- Treatment sum of squares
- The part of the total variation explained by differences between the group means, computed as each group's size times its squared distance from the grand mean.
- Residual sum of squares
- The part of the total variation left inside the groups, computed from each observation's distance from its own group mean. Divided by its degrees of freedom it becomes the shared variance estimate.
- Mean square
- A sum of squares divided by its own degrees of freedom. Mean squares do not add to a total, which is one difference from the sums of squares above them.
- F ratio
- The treatment mean square divided by the residual mean square. It is written numerator first and its distribution takes an ordered pair of degrees of freedom.
- Contrast
- A weighted combination of group means whose weights sum to zero. A pairwise difference is the simplest example, and any contrast can be tested with the shared variance estimate.
- Multiplier
- The constant an interval is built with. Different multiplier methods provide simultaneous coverage across different families of comparisons.
- Union bound
- The fact that the probability of at least one of several events is at most the sum of their probabilities. It is the entire argument behind the Bonferroni multiplier, and being a bound is what makes it conservative.
- Simultaneous coverage
- The probability that all the intervals in a family contain their true values at once, as opposed to the coverage of any one of them.
- Data snooping
- Choosing which comparisons to make after seeing the results. It invalidates a correction designed for a family fixed in advance, and one multiplier method is built specifically to permit it.
- Residual
- An observation minus its own group mean. Pooling all of them into one plot is a more reliable way to check normality than examining each small sample separately.
One-way ANOVA and post hoc comparisons FAQ
Why not just run a t test on every pair of groups?
Two reasons. The comparisons multiply, so the chance of at least one false positive climbs with the number of groups, which grows quadratically. And each pairwise test estimates the spread from only the two groups it involves, throwing away the information in the others.
The analysis of variance asks one question once and estimates the spread from every observation, which makes the follow up comparisons sharper than standalone ones would have been.
What does rejecting the overall test actually tell me?
That the group means are not all equal, and nothing at all about which of them differ from which. The alternative hypothesis is that at least one differs, which is deliberately vague, and the vagueness is what makes the follow up necessary. A conclusion naming particular groups on the strength of the F test alone has claimed something the test cannot deliver.
Which multiplier method should I use?
Read what each one permits before comparing their sizes. Bonferroni is valid for a list of contrasts committed to in advance. Tukey is built for the all pairwise family and is exact when the sample sizes are equal, conservative when they are not. Scheffe is exact for the family of all possible contrasts, which is why it is the one that permits choosing a comparison after looking at the means.
Selecting the method after seeing which gives the smallest p-value is not a choice between them.
Is Bonferroni always the most conservative?
No, and generalising from three groups is a trap. With few comparisons Bonferroni is competitive and can even give a smaller multiplier than Scheffe. As the number of comparisons grows it becomes markedly the worst of the three, because its penalty scales with that number, while Scheffe does not change at all since it already covers every contrast. Tukey is the smallest for the all pairwise family in both regimes.
How do I check the assumptions?
On the residuals rather than on each sample, because pooling them gives one picture with all the observations in it instead of several unreliable small ones. A normal quantile plot of the residuals checks normality, and residuals against fitted values check the common variance assumption, with a widening funnel as the classic failure.
The unit teaches no formal test for either, so the check is graphical and the judgement is yours.
Exam move
Draw the decomposition once, as a single bar splitting into two, and label every quantity on it: the two sums of squares, the two degrees of freedom, the two mean squares and the ratio. If you can reproduce that picture you can fill in any table you are given, and filling in a partially completed table is one of the most predictable question types in this material.
Practise it with the two arithmetic checks always in place, that the sums of squares add and the degrees of freedom add, and practise writing the F ratio numerator first, because an inverted pair of degrees of freedom is a silent error.
Second, build a three row summary of the multiplier methods with a column for what each is exact for and a column for what each permits afterwards, and revise the second column harder, since the scope of use is the examinable part and the numbers are not.
Third, learn the four assumption failures with their matched repairs as pairs rather than as two lists, because a question that names a failure is asking for the specific response to that failure and not for a survey of the alternatives.
Working through One-way ANOVA and post hoc comparisons in DATA2002? Sia is AskSia’s AI Statistics tutor — ask any DATA2002 One-way ANOVA and post hoc comparisons question and get a clear, step-by-step explanation grounded in how DATA2002 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
