BB2025 Chap.8 Reliability, Validity and Diagnostic Accuracy
Reliability, Validity and Diagnostic Accuracy
Consistency does not guarantee a correct target
Reliability concerns repeatability under specified raters, occasions, instruments and populations. Validity concerns whether evidence supports the intended score interpretation and use. An instrument can produce tightly clustered results that are systematically displaced.
Reliability is necessary for many valid uses, but it cannot prove construct alignment, clinical meaning or appropriate classification.
Translate reliability into score units
Relative coefficients such as the intraclass correlation depend partly on between-person variability. The standard error of measurement expresses error in the instrument's units.
A minimum detectable difference expands that error for change between two measurements at a chosen confidence level. Detectability remains separate from clinical importance, so an observed change can satisfy either, both or neither threshold.
Diagnostic meaning depends on conditioning
Sensitivity and specificity condition on true status.
Predictive values condition on the observed test result and therefore change with prevalence. Moving a continuous threshold trades sensitivity against specificity. ROC analysis summarises ranking discrimination across thresholds but does not choose the clinical cutoff or establish calibration. Reference-standard quality and verification procedures remain part of the accuracy claim.
What this chapter covers
- 01
Internal consistency and item behaviour
- 02
Test-retest and inter-rater designs
- 03
Intraclass correlation selection
- 04
Standard error of measurement
- 05
Minimum detectable difference
- 06
Construct and criterion validity
- 07
Sensitivity, specificity and predictive values
- 08
ROC curves and threshold decisions
Worked example · free
Compare change with measurement error
- 3Substitute the standard error into the formula.
- 2Compute and compare the threshold.
- 3Separate detectability from clinical importance.
Key terms
- Internal Consistency
- The degree to which items behave as a coherent scale within a specified sample and administration.
- Intraclass Correlation
- A relative reliability coefficient comparing between-unit variation with total variation under a specified design.
- Measurement Error
- Variation in observed scores that does not represent the underlying construct intended by the measurement.
- Detectable Difference
- A change threshold derived from measurement error for a stated confidence level and score use.
- Predictive Value
- The probability of condition status among people with a particular observed test result in a setting.
- ROC Curve
- A plot of sensitivity against one minus specificity as a diagnostic threshold changes.
Reliability, Validity and Diagnostic Accuracy FAQ
Can a measure be reliable but invalid?
Yes. Repeated measurements can cluster closely around a biased target, or a coherent item set can measure the wrong construct. Reliability limits how well a score can support many uses, but validity requires additional evidence from theory, relationships, known groups, reference measures and the actual decision context.
Why is correlation not enough for agreement?
Correlation shows whether two measurements preserve ordering, not whether they produce the same values. One rater can score every participant several units higher and still correlate almost perfectly with another. Agreement requires examining differences, systematic bias, limits of agreement and a reliability form matched to whether absolute values or ranks matter.
Why do predictive values change across settings?
They condition on the test result, so the proportion of true cases among everyone tested matters. In a low-prevalence screening population, false positives may outnumber true positives even when sensitivity and specificity are stable. Report the target setting and prevalence rather than transporting predictive values from a specialist clinic without qualification.
Exam move
Build a comparison table with reliability, agreement, validity and diagnostic accuracy as separate columns. For each coefficient, write the design that produces it, its unit or scale and the decision it can support. Recompute one minimum detectable difference from a printed SEM, then draw a two-by-two diagnostic table and label every denominator.
Practise threshold questions by predicting sensitivity and specificity direction before looking at an ROC curve. Add a column for the consequence of error: group research, individual monitoring, screening or diagnosis. The acceptable precision changes with that use. Rework one prevalence example by keeping sensitivity and specificity fixed while reducing the number of true cases, and watch positive predictive value fall.
This secures the conditioning more effectively than memorising four formula names. Pair every coefficient with a plot: differences for agreement, score distributions for thresholds and observed versus expected values for calibration. The picture should test the claim the number compresses. For every diagnostic scenario, name the decision made after a positive result and after a negative result.
Then identify which error is costlier in that setting. Threshold choice becomes defensible only when it is linked to those consequences.
Working through Reliability, Validity and Diagnostic Accuracy in BB2025? Sia is AskSia’s AI Statistics tutor — ask any BB2025 Reliability, Validity and Diagnostic Accuracy question and get a clear, step-by-step explanation grounded in how BB2025 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
