PUBH5010 · Epidemiology Methods And Uses
Critical Appraisal
The long Section-C question hands you a realistic study and asks: can I believe this result? Critical appraisal is the repeatable pipeline that answers it, and the exam reward is structure, not free-form criticism. The walk is always the same: identify the study design and the hypothesis it tests; ask whether chance could explain the finding (precision, confidence interval, p-value) and whether it is statistically and clinically meaningful; then work the three validity threats in order — selection (was the sample a fair window on the study base?), measurement / information (were exposure and outcome measured well, and is any misclassification differential?), and confounding (is a third variable mixed in, and which way?). Each threat is named, its likely direction argued, and its importance weighed — because the marks go to saying which flaw matters most and how it changes your reading of the result, not to listing every conceivable problem. Finish with a calibrated bottom line: how much the association can be trusted, and what would strengthen the evidence. This pipeline is just chapters 5–7 (selection, confounding, measurement) reassembled into one disciplined answer.
What this chapter covers
- 01Framing the question: design, hypothesis, and the measure reported
- 02Could chance explain it? Precision, confidence intervals, p-values
- 03Selection: is the sample a fair window on the study base?
- 04Measurement / information: validity and the direction of misclassification
- 05Confounding: the three criteria and the direction of bias
- 06Weighing which flaw matters most
- 07A calibrated bottom line and what would strengthen the evidence
Worked example: appraising a case-control study in the exam order
- +1Design & hypothesis. Case-control testing a diet–disease link; the measure is an odds ratio, which is correct for this design.
- +1Chance. The 95% CI (1.2–7.5) excludes 1, so the result is statistically significant, but it is wide — the true OR is imprecise and could be modest or large.
- +1Selection. Controls are other hospital patients, who may have diets unlike the source population — potential selection bias of unclear direction; ask whether their conditions relate to the exposure.
- +2Measurement. Exposure is by recall, and cases may recall diet differently from controls — recall bias (differential), which here likely inflates the OR.
- +1Confounding & bottom line. Diet tracks other lifestyle factors (a confounder if unadjusted). On balance the association is probably real (CI excludes 1) but overstated by recall bias and possible confounding; a cohort with measured diet would strengthen it.
Key terms
- Critical appraisal
- The structured evaluation of whether a study's reported association can be believed, walking design, chance, selection, measurement and confounding in a fixed order, naming each threat, its direction and its importance, and ending in a calibrated conclusion.
- Internal validity
- Whether the study correctly measured the association within its own sample — i.e. is free enough of chance, selection bias, information bias and confounding. It must be established before any question of generalisability.
- External validity
- Whether the study's finding generalises beyond its sample to other populations or settings. It only matters once internal validity holds: a biased estimate is not worth generalising.
- Role of chance
- Whether random variation could explain the result, judged from precision — the width of the confidence interval and the p-value. A statistically significant result can still be clinically trivial, and a non-significant one can reflect low power, so both significance and magnitude are appraised.
- Bottom line (calibrated conclusion)
- The appraisal's closing judgement: how much the association can be trusted given the flaws found, the net likely direction of bias, and what design or measurement change would strengthen the evidence.
Critical Appraisal FAQ
What order should I appraise a study in?
Use a fixed pipeline so you never miss a step under time pressure: name the design and hypothesis; assess chance (confidence interval, p-value, precision); then the three validity threats — selection bias, measurement/information bias, and confounding — each with its likely direction; then weigh which matters most and give a calibrated bottom line. The fixed order is itself worth marks because it shows disciplined reasoning.
Do I need to list every possible flaw?
No — and a long unranked list scores poorly. The marks go to identifying the flaws that plausibly apply to this study, arguing the direction each would push the estimate, and judging which one matters most for interpreting the result. Depth on the decisive flaws beats breadth on hypothetical ones.
What's the difference between internal and external validity?
Internal validity asks whether the association is correct within the study's own sample (free of chance, bias and confounding). External validity asks whether it generalises to other populations. Internal comes first: there is no point generalising a result that is biased to begin with, so the appraisal pipeline is really an internal-validity check, with generalisability considered last.
How do I judge the role of chance?
Read the confidence interval and p-value. A 95% CI that excludes the null (1 for ratios, 0 for differences) signals statistical significance; its width signals precision — a wide interval means the true value is poorly pinned down even if significant. Always pair this with clinical significance: a tight, significant interval around a trivial effect may not matter, and a wide interval may reflect low power rather than no effect.
Exam move
Internalise one fixed pipeline and run it on every Section-C study: design & hypothesis → chance (CI/precision) → selection → measurement → confounding → bottom line. For each threat, name it, argue its direction, and weigh its importance — then explicitly say which flaw matters most and how it changes your trust in the result. This chapter is chapters 5–7 reassembled, so revise them together; and always close with a calibrated conclusion plus the one design change that would strengthen the evidence.
