RS517 Chap.5 Hypothesis Tests, Errors and Power
Hypothesis Tests, Errors and Power
A decision under a model
The taught sequence states the null, sets alpha, chooses and computes a statistic, obtains a p-value and compares it with alpha. The outcome is to reject or not reject the null, never to prove a hypothesis true.
Errors trade against information
Type I error rejects a true null, while Type II error fails to reject a false null.
Lowering alpha at fixed information reduces the first risk and increases the second. Larger samples, reliable outcomes and efficient designs can improve power without the same trade-off.
Intervals preserve scale
A p-value describes compatibility with a null model, not effect magnitude or the probability the null is true. Confidence intervals show the estimate and precision on the clinical scale.
Multiplicity requires a defined family of tests and a control strategy matched to confirmatory or exploratory aims.
Define the parameter before the hypothesis
A null hypothesis concerns a parameter under a model, such as a mean difference of zero or an odds ratio of one. It is not automatically the claim that nobody benefits, the distributions are identical or the treatment is clinically useless.
The alternative should reflect the scientific question. A one-sided form is defensible only when an effect in the opposite direction would not alter the decision. State the effect scale before reading the test result.
Interpret estimates, intervals and p-values together
The p-value measures compatibility between the observed statistic and the null model, conditional on assumptions.
It does not give the probability that the null is true and does not quantify effect size. A confidence interval displays effects compatible with the procedure and data; compare it with zero and a clinically important threshold.
A narrow biased interval remains wrong, while a wide interval can honestly reveal that the study cannot distinguish harm, trivial benefit and meaningful benefit.
Power is a design property, not a post hoc excuse
Power depends on the specified effect, variability, sample size, allocation, measurement reliability and analysis.
A Type I error rejects a true null under the model; a Type II error misses the specified alternative. Increasing sample size can reduce random uncertainty but cannot remove confounding or measurement bias. Post hoc power calculated from the observed estimate usually restates the p-value and adds little.
Discuss precision and the effects still compatible with the interval instead.
Revision drill
Take an interval that crosses zero and a clinical threshold. Write the statistical decision, the range of clinically plausible effects and the feature most likely to improve information. Then change only sample size, only measurement reliability and only the assumed target effect.
Explain how each change affects planned power and what it cannot repair.
What this chapter covers
- 01
Null and alternative hypotheses
- 02
Alpha and the testing sequence
- 03
Type I and Type II errors
- 04
Power and sample information
- 05
Confidence intervals and clinical thresholds
- 06
Multiplicity and analysis families
Worked example · free
Interpret an inconclusive interval
- 3State the statistical decision without accepting the null.
- 5Interpret the interval against zero and the clinical threshold.
Key terms
- Null Hypothesis
- The specified model claim tested against the observed statistic.
- Type I Error
- Rejecting a true null hypothesis under the stated model.
- Type II Error
- Failing to reject the null when the specified alternative is true.
- Power
- The probability of rejecting the null under a specified alternative and design.
- Confidence Interval
- A range produced by a procedure calibrated to cover the target parameter at a stated long-run rate.
Hypothesis Tests, Errors and Power FAQ
What does p less than 0.05 mean?
Under the null model and analysis assumptions, a result at least as incompatible with that null as the observed result would occur less than five per cent of the time. It does not mean the null has less than five per cent probability or that the effect is important.
Can a non-significant result prove equivalence?
No. Failure to reject zero can reflect a small effect or imprecision. Equivalence requires prespecified acceptable margins and a design capable of excluding effects outside them. Examine the interval against those margins instead of treating the significance label as evidence of sameness.
How can power be increased?
Increase information through sample size, reliable measurement, balanced allocation, repeated measures used efficiently or stronger design control. Raising alpha also increases the rejection probability but accepts more Type I error. Power must be planned for a meaningful effect and plausible variability before the data are seen.
Exam move
Draw the truth-by-decision grid until Type I and Type II errors are automatic. For every result, write five clauses in order: estimate, confidence interval, p-value decision, clinical threshold and design limit. Practise correcting probability reversals and any sentence that accepts the null.
Working through Hypothesis Tests, Errors and Power in RS517? Sia is AskSia’s AI Statistics tutor — ask any RS517 Hypothesis Tests, Errors and Power question and get a clear, step-by-step explanation grounded in how RS517 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.