DATA2002 Chap.6 Testing means with the t-test family
Testing means with the t-test family
Module 1 counted observations into categories. From here the response is a measurement, and the first question about a measurement is almost always about its mean. The unit builds the machinery once and then reuses it, so it is worth learning the general shape rather than four separate formulas: every test statistic in this part of the unit is an estimator minus the parameter it estimates, divided by its standard error.
For that ratio to follow a t distribution, two conditions are needed. The estimator on top must be normally distributed, and the standard error underneath must behave like the true standard deviation times the square root of a scaled chi-squared quantity, independently of the numerator.
Replacing the unknown population standard deviation by the sample value is exactly what turns a normal statistic into a t statistic, and the extra uncertainty that substitution introduces goes into the tails, which is why a t critical value always sits further from zero than the corresponding normal one. Three designs then follow from one question: how the observations relate to each other.
One sample against a stated value uses the sample size minus one degrees of freedom. Two independent samples come in two versions, one pooling the variances when a common variance is assumed and one refusing to pool, which is the default in software and produces degrees of freedom estimated from the data and usually not a whole number.
Paired measurements are reduced to a single sample of differences, and the degrees of freedom count pairs rather than measurements. That last distinction is the most consequential in the module, because the same twenty two numbers can give opposite verdicts depending only on whether the design is paired.
What this chapter covers
- 01
The general shape: estimator minus parameter over standard error, and the two conditions it needs
- 02
Where the t distribution comes from, and why its tails are heavier than a normal's
- 03
Estimator, estimate and standard error, and why the unit keeps the three words apart
- 04
The one sample test, its statistic and its degrees of freedom
- 05
The pooled two sample test, and the weighted average that produces the pooled variance
- 06
The unequal variance version, its approximate degrees of freedom, and why it is not a proper t statistic
- 07
Which version software gives you when you say nothing
- 08
When pooling is safe and the configuration in which it is anti-conservative
- 09
Paired data reduced to one sample of differences
- 10
Degrees of freedom counted in pairs, and the direction of the error when they are not
- 11
The same numbers analysed both ways, and why the paired analysis is sharper
- 12
A four question selection map from design to test
Identify a mis-specified analysis and say which way it errs
- +1.5The design is paired: each trainee supplies both times, so the two columns are linked trainee by trainee and are not two independent samples.
- +1.5The correct analysis differences the two times within each trainee and runs a one sample test on the eighteen differences, with seventeen degrees of freedom rather than thirty four.
- +1.5The reported p-value is too large, because the unpaired denominator includes the variation between trainees, which the paired analysis removes from the comparison entirely.
- +0.5The consequence is a conclusion of no difference that the correct analysis may well reverse. This is a mistake that hides real effects rather than inventing them, which makes it easy to miss.
Key terms
- Estimator
- The random variable that produces an estimate, written with a capital letter. It is conceptually distinct from the number it took on in this sample, which is the estimate.
- Standard error
- The standard deviation of an estimator. When the population standard deviation is unknown it is estimated from the sample, and that substitution is what makes the statistic follow a t distribution.
- Pooled variance
- A single variance estimate formed by combining two sample variances, weighted by their degrees of freedom. It is only appropriate when both populations can be assumed to share a variance.
- Welch adjustment
- The version of the two sample test that estimates each variance separately and never pools, at the cost of approximate degrees of freedom that need not be a whole number.
- Paired design
- A design in which each observation in one group is matched to exactly one in the other, so the analysis proceeds on the within pair differences.
- Degrees of freedom
- The parameter indexing the t distribution, equal to the sample size minus the number of parameters estimated in the denominator. For a paired test it counts pairs, not measurements.
- Two sided alternative
- An alternative hypothesis that the parameter differs from the null value in either direction, requiring the significance level to be split between two tails.
- Common variance assumption
- The assumption that two populations share a variance, which licenses pooling. It is an assumption about populations, not an observation about the two sample standard deviations.
- Independent samples
- Two sets of observations with nothing linking a member of one to a member of the other. Their sizes need not be equal, which is one visible difference from a paired design.
- Sample variance
- The average squared deviation from the sample mean, divided by the sample size minus one. The divisor accounts for the mean having been estimated from the same data.
Testing means with the t-test family FAQ
How do I tell a paired design from two independent samples?
Ask whether each observation in one group is matched to exactly one in the other by something in the design. The same subject measured before and after, two measurements on the same item, litter mates, the left and right of one object: all paired. If the two groups are simply two sets of units with nothing linking member to member, they are independent, and one visible clue is that their sizes need not be equal.
A paired design with unequal group sizes is a contradiction and usually indicates that something has been lost.
Which version of the two sample test should I use?
Unless there is a good reason to believe the two populations share a variance, use the version that does not pool. It is the default in software, it costs very little when the variances really are equal, and it protects you when they are not.
Pooling is most dangerous in one specific configuration, when the larger variance belongs to the smaller sample, because the pooled estimate is then pulled towards the variance from the larger sample, the standard error is understated and the test rejects too often.
Why are the degrees of freedom sometimes not a whole number?
Because the unequal variance version estimates them from the data. Its denominator is not a scaled chi-squared quantity independent of the numerator, so the statistic is not a proper t statistic and its distribution is only approximately a t distribution with degrees of freedom that must be estimated. A printed figure such as 18.45 is telling you which version was run, and describing that output as a pooled test misreads it.
What does the normality assumption actually require?
That the population from which the observations are drawn is normal, not that the sample looks normal. On a large sample the requirement is close to free, because the sample mean is approximately normal whatever the population, but on a dozen observations it is doing real work and cannot be waved through.
The check is graphical: a normal quantile plot of the data, or of the differences in a paired design, looking for systematic curvature or a point far off the line rather than for perfection.
What if the normality assumption fails?
Then the answer is not to run the test anyway. Two routes follow in the next chapters: replace the values by their ranks, which discards magnitude information in exchange for immunity to shape, or build the null distribution directly from the data by permuting.
Which route to prefer depends on how badly the assumption fails and on what the question asks, and the decision should be made from the diagnostic plot before any p-value is seen.
Exam move
Learn the general shape first and the four formulas second, because the shape is what lets you reconstruct a statistic you half remember: something estimated, minus what it estimates under the null, over its standard error. Then build a four row table of your own with the designs down the side and three columns, the statistic, the degrees of freedom, and the phrase in a question that signals it.
The degrees of freedom column is where marks are lost, so write it out rather than assuming you will remember, and pay particular attention to the paired row, since counting measurements instead of pairs inflates them and produces a p-value that is too small. Second, practise the routing rather than the arithmetic: take scenarios, decide the design, and stop.
Third, get into the habit of writing the assumption line as something to be checked rather than declared, naming what you would plot and what you would look for, because that is how the mark is worded. Finally, work at least one example where the same numbers are analysed as paired and as independent, so that the size of the difference is something you have seen rather than something you have been told.
Working through Testing means with the t-test family in DATA2002? Sia is AskSia’s AI Statistics tutor — ask any DATA2002 Testing means with the t-test family question and get a clear, step-by-step explanation grounded in how DATA2002 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
