91828 Chap.6 P-values, Confidence and Clinical Meaning
P-values, Confidence and Clinical Meaning
P-values describe data under a null model; they do not state the probability that a hypothesis is true or the size of an effect. This chapter makes the confidence interval the centre of interpretation, comparing its values with both the correct null and a justified minimum clinically important difference. It also covers multiplicity, equivalence claims and the difference between statistical precision and freedom from bias.
What this chapter covers
- 01
Null hypothesis
- 02
P-value meaning
- 03
Confidence intervals
- 04
Clinical importance threshold
- 05
Multiplicity
- 06
Equivalence and non-inferiority
Worked example · free
Read an interval against two thresholds
- +1The point estimate favours treatment by 6 points and exceeds the 4-point MCID.
- +1The interval excludes the null difference of zero.
- +1Its lower limit of 2 lies below the MCID, so a clinically small effect remains plausible.
- +1Its upper limit permits a large 10-point benefit, showing substantial remaining range.
- +1The average result is promising but does not establish that the true benefit reaches the chosen clinical threshold.
Key terms
- Null Hypothesis
- A null hypothesis specifies the no-effect or reference model tested by an analysis.
- P-value
- A p-value measures how incompatible the observed data are with a specified null model and assumptions.
- Confidence Interval
- A confidence interval shows a model-based range of effect values compatible with the observed data.
- Clinical Threshold
- A clinical threshold defines an effect magnitude considered important enough to matter for a decision.
P-values, Confidence and Clinical Meaning FAQ
Is p below 0.05 the probability the result is true?
No. A p-value is calculated assuming the null model and measures the extremeness of the data under that model. It does not provide the probability of either hypothesis, the probability of replication or the clinical importance of the effect.
Can a significant result be clinically trivial?
Yes. A precise small effect can exclude the null while lying below a justified threshold for patient importance. Report the estimate and interval, compare them with the MCID, and include harms, burden and preferences before making a decision.
Does a non-significant result show equivalence?
No. It may simply be imprecise, with an interval spanning benefit and harm. Equivalence or non-inferiority requires prespecified margins, suitable design and analysis, and an interval interpreted against those margins rather than failure to cross a significance cutoff.
What does a 95% confidence level mean?
Under repeated use of the same procedure and assumptions, 95% of constructed intervals would contain the target parameter. It does not mean this realised interval has a 95% probability of containing a fixed true value. Use the interval to describe compatible effects, not posterior probability.
How should subgroup findings be interpreted?
Look for a prespecified subgroup, a plausible rationale and a direct interaction estimate with uncertainty. A significant result in one group and a non-significant result in another does not itself establish different effects. Exploratory findings usually motivate confirmation rather than immediate practice change.
Why can a narrow interval still mislead?
An interval represents random uncertainty under its model, not systematic error. Selection bias, switched outcomes, unblinded measurement or an inappropriate analysis can shift the estimate without widening it. Appraise the design and analysis population before treating precision as credibility.
Assessment move
Build a set of confidence-interval cards that vary effect measure, direction and relation to clinical thresholds. On the front, write only the point estimate, interval, outcome and minimum clinically important difference. On the back, write five clauses: magnitude and units, plausible range, relation to the null, relation to the clinical threshold, and the principal validity or applicability limit.
Include differences with a null of zero and ratios with a null of one. Use examples where the interval excludes the null but stays below the MCID, spans both no effect and important benefit, or lies entirely beyond the clinical threshold. This removes the habit of treating statistical significance as the final interpretation.
For p-values, rehearse the definition under the null model and list claims it cannot support: probability the null is true, effect size, importance or replication probability. Compare results just below and above 0.05 using their estimates and intervals so an arbitrary cutoff does not become a cliff.
Add multiplicity exercises that identify primary, secondary and exploratory outcomes, prespecified subgroups and repeated time points. Ask what error-control strategy or interaction test is required rather than counting small p-values. Read a trial registration beside its report to detect switched outcomes or newly prominent subgroups.
Pair every narrow interval with a bias audit; precision does not protect against selection, differential missingness or unblinded measurement. Pair every wide interval with a decision statement naming the important effects still possible. For AT1, replace the word significant with the actual estimate and range, then state clinical meaning. For AT2, justify the MCID before results exist and connect it to sample-size planning.
End by explaining an inconclusive interval to a patient in plain language without saying that treatment has no effect. The goal is a bounded decision, not a binary label.
