BIOS6902 Chap.7 Hypothesis Testing: Errors, Power and Neyman-Pearson
Hypothesis Testing: Errors, Power and Neyman-Pearson
Eight cups, and the whole apparatus
The story the course opens with concerns a colleague of Fisher who insisted she could taste which liquid had gone into the cup first. Eight cups were prepared, four each way, and presented in a shuffled order; she picked out all four of one kind without error.
Guessing would produce that outcome once in seventy attempts, about fourteen in a thousand, and since that sits below one in twenty the result counted as significant. Every piece of modern testing is already there: a claim of no ability, a statistic, a probability worked out under that claim, and a line drawn in advance.
Formally, a test opposes a null hypothesis to an alternative, with the two parameter sets disjoint.
A simple hypothesis names a single value; a composite one names more. You choose a test statistic and a rejection region, and the test is the indicator of that region. Two mistakes are possible and they are not symmetric in consequence: rejecting a true null, and failing to reject when the alternative holds.
The power function records the probability of rejection at every parameter value and therefore describes both mistakes at once.
Size, level, and a distinction that gets examined
A test has a given size when the worst-case probability of false rejection across the null set equals that number, and a given level when the worst case is merely bounded by it.
Every test of a given size has that level; the converse fails whenever the supremum is not attained. With discrete data this is routine rather than exotic: achievable rejection probabilities come in jumps, usually no region has probability exactly one in twenty, and the resulting test is mildly conservative.
Saying so explicitly when a question involves a small binomial sample is a cheap mark.
Against a simple alternative there is an optimal answer. The Neyman-Pearson lemma says the test that rejects for large values of the likelihood ratio is the most powerful test at that level against that particular alternative. The result is elegant and narrow.
Real alternatives are almost never a single point, because a clinician asks whether a drug works rather than whether its effect is exactly some specific number, so a test optimal against one alternative may be poor against the next.
The two general-purpose procedures do the actual work.
The two statistics you will actually use
A Wald test divides the distance between the estimate and the null value by a standard error and compares the result with a normal critical value.
It is cheap, it applies wherever asymptotic normality does, and its main subtlety is which standard error belongs in the denominator: under the null's own logic the null value goes under the square root, whereas an interval uses the estimate.
A likelihood-ratio test compares the largest achievable likelihood under the null with the largest achievable overall, and rejects when that ratio is small.
Wilks' theorem is what makes the ratio usable for composite hypotheses.
Under regularity and under the null, twice the negative logarithm of the ratio converges to a chi-square distribution whose degrees of freedom equal the difference in dimension between the unrestricted and restricted parameter sets. Degrees of freedom count constrained parameters, not observations, and getting that count wrong is the most common technical error in answers on this material.
Applied to a rate, the two procedures usually point the same way at moderate sample sizes, and comparing them on the same data is worth doing precisely because agreement cannot be assumed in general.
What this chapter covers
- 01
The tea-tasting experiment as the original test
- 02
Null and alternative, simple and composite
- 03
Type I and Type II error, and the power function
- 04
Size against level, and why discrete data separate them
- 05
The Neyman-Pearson lemma and its narrow scope
- 06
Wald tests, and which standard error belongs in the denominator
- 07
Likelihood-ratio tests and Wilks' theorem
Worked example · free
Finding a critical value and then the power it delivers
- 2State the distribution of the sample mean under the null.
- 2Solve for the threshold that gives the required size.
- 2Recompute the distribution under the alternative and obtain the power.
- 2Take the limit in the sample size and interpret it.
Key terms
- Null Hypothesis
- The statement about the parameter that a test is designed to challenge, usually expressing no effect or no difference.
- Power Function
- The probability of rejecting the null, recorded at every parameter value, so it describes false rejection and successful detection at once.
- Type I Error
- Rejecting a null hypothesis that is in fact true. Its worst-case probability across the null set is what a size or level constrains.
- Size
- The largest probability of false rejection over the null set, attained rather than merely bounded.
- Level
- An upper bound on the probability of false rejection over the null set, which may not be achieved exactly when the statistic is discrete.
- Neyman-Pearson Lemma
- The result that, for a simple null against a simple alternative, the likelihood-ratio test is most powerful among all tests of a given level.
- Wald Statistic
- The distance between an estimate and a null value divided by a standard error, compared with a normal critical value.
- Wilks' Theorem
- The result that twice the negative log likelihood ratio has an approximate chi-square distribution under the null, with degrees of freedom equal to the dimension reduction.
Hypothesis Testing: Errors, Power and Neyman-Pearson FAQ
Why is the Type I error controlled rather than the Type II error?
Because the two mistakes fall on different parties and carry different costs. Rejecting a true null means announcing an effect that does not exist, which propagates into practice, into later research and into decisions; the convention fixes that rate in advance and then buys as much detection ability as the budget allows. It is a convention about who bears the risk, not a mathematical necessity.
Which standard error goes under the square root in a Wald test?
The one implied by the hypothesis you are assuming. A test computes a probability under the null, so the null value belongs in the denominator; an interval assumes no particular value, so the estimate belongs there. Both forms of the test statistic exist in the literature and both are defensible, but mixing them inside one calculation is not.
How do I count the degrees of freedom for a likelihood-ratio test?
Subtract the dimension of the restricted parameter set from the dimension of the unrestricted one. It counts how many free parameters the null gives up, never how many observations you have. Testing a single mean against a fixed value costs one dimension; testing whether two groups share a two-parameter distribution costs two.
Is the Neyman-Pearson lemma useful in practice?
Directly, rarely: it requires the alternative to be a single specified value, which real questions almost never supply. Indirectly, constantly: it explains why likelihood ratios appear everywhere in testing, and the general likelihood-ratio test is the natural extension of it to composite hypotheses.
Exam move
Write out the four-line answer skeleton and use it every time: both hypotheses in parameter terms, the statistic and its null distribution, the rejection region or the p-value, then the decision with its scope. Most marks in a testing question are for the fourth line, which is the one students leave out.
Working through Hypothesis Testing: Errors, Power and Neyman-Pearson in BIOS6902? Sia is AskSia’s AI Statistics tutor — ask any BIOS6902 Hypothesis Testing: Errors, Power and Neyman-Pearson question and get a clear, step-by-step explanation grounded in how BIOS6902 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
