BB2025 Chap.4 Hypothesis Tests and Statistical Decisions
Hypothesis Tests and Statistical Decisions
Read the test as a conditional argument
A hypothesis test assumes a null model, standardises an observed effect against expected variability and calculates how unusual that statistic or something more extreme would be under the reference model. The resulting p value does not assign probability to the hypothesis.
It also does not show effect size, precision or clinical importance.
Join decision to estimation
Report the estimated effect with direction and units, then a confidence interval and the test decision. The interval displays a range of population values compatible with the estimate and model.
A small p value can accompany a clinically trivial effect in a large sample, while a useful effect can remain uncertain in a small sample. Meaning therefore comes from magnitude and context rather than the threshold alone.
Plan error trade-offs
Alpha controls the long-run Type I error rate of a testing procedure. Power is the chance to reject the null for a specified true effect and equals one minus beta.
Lowering alpha without increasing information reduces power. Sample-size planning makes these trade-offs explicit through the target effect, variability, alpha, power and expected attrition.
What this chapter covers
- 01
Null and alternative hypotheses
- 02
One-sided and two-sided tests
- 03
Test statistics and reference distributions
- 04
Conditional p value interpretation
- 05
Effect estimates and confidence intervals
- 06
Statistical and clinical importance
- 07
Type I and Type II errors
- 08
Power and sample-size planning
Worked example · free
Interpret a result against an importance threshold
- 2State the estimate and direction.
- 3Interpret the interval and p-value decision.
- 3Compare with clinical importance without overclaiming.
Key terms
- Test Statistic
- A standardised summary comparing the observed effect with variability expected under a reference model.
- P Value
- The probability under the null model of a statistic at least as extreme as the observed one.
- Type I Error
- Rejection of a true null hypothesis under the long-run decision procedure.
- Type II Error
- Failure to reject a false null hypothesis for a specified alternative effect.
- Statistical Power
- The probability that a study rejects the null for a specified true effect and design.
- Effect Estimate
- A sample-based numerical estimate of the magnitude and direction of a population relationship or difference.
Hypothesis Tests and Statistical Decisions FAQ
Does a small p value show that the null is unlikely?
Not from a standard frequentist test. The p value conditions on the null model and asks about the observed statistic or something more extreme. It does not reverse that relationship to calculate a probability for the hypothesis. To interpret the result, add the effect estimate, confidence interval, study design and clinical context.
What does a non-significant result mean?
It means the data did not cross the selected rejection threshold under the model. It does not establish no effect or equivalence. Inspect the confidence interval: a wide range spanning important benefit and harm is inconclusive, while a narrow interval entirely inside a pre-specified negligible range may support a separate equivalence conclusion.
Why does lowering alpha reduce power?
A lower alpha requires a more extreme statistic before rejection, which reduces false positives under a true null. With the same sample and true effect, fewer results cross that stricter boundary, so Type II error rises and power falls. Additional information, commonly a larger sample or more precise design, is needed to restore power.
Exam move
Practise translating output into one disciplined paragraph. Start with the comparison and population quantity, state the estimate with units and direction, give the confidence interval, then describe the p-value decision without assigning probability to the null. Finish with clinical meaning and the strongest design limitation.
Make error cards that show the condition required for Type I error, Type II error, power and confidence, because most mistakes reverse which truth or model the probability assumes. For every result, draw a number line with zero, the interval endpoints and any clinical threshold. Say whether the data exclude no effect, trivial effects and important effects separately.
Then change the sample size mentally while holding the estimate constant and predict how precision and the decision might move without changing practical magnitude. Correct one false interpretation aloud after every calculation. Speaking the conditional phrase under the null makes probability reversal easier to notice than silent rereading. Create paired cards showing the same estimate with wide and narrow intervals.
Decide what each permits about direction and clinical importance before viewing a p value. Precision should change the warranted conclusion, not the observed effect itself.
Working through Hypothesis Tests and Statistical Decisions in BB2025? Sia is AskSia’s AI Statistics tutor — ask any BB2025 Hypothesis Tests and Statistical Decisions question and get a clear, step-by-step explanation grounded in how BB2025 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
