RS517 Chap.9 Reliability, Validity and Diagnostic Accuracy
Reliability, Validity and Diagnostic Accuracy
Consistency is not correctness
Reliability concerns consistency across occasions, raters or forms. Validity concerns whether evidence supports the intended interpretation and use.
A stable biased measure can be highly reliable and invalid for its purpose.
Choose the coefficient from the use
Test-retest, intra-rater, inter-rater, alternate-form and internal-consistency questions require different designs. Correlation shows association but not absolute agreement.
ICC form, kappa, standard error of measurement and agreement plots should be chosen from whether ranking, interchangeability or individual change matters.
Diagnostic denominators
Sensitivity conditions on people with the reference condition and specificity on those without it. Predictive values condition on the test result and change with prevalence.
ROC curves show the sensitivity and false-positive trade-off as a threshold moves.
Choose reliability for the intended decision
Reliability is consistency under specified conditions, so the form must match the use. Test-retest concerns stability across time, inter-rater and intra-rater forms concern scoring conditions, and internal consistency concerns relationships among items.
Correlation alone can remain high when one rater is systematically higher. Name whether consistency or absolute agreement is required, whether a single rating or average will be used, and the population in which the coefficient was estimated.
Accumulate validity evidence instead of seeking one badge
Content evidence asks whether the construct domain is represented.
Criterion evidence compares with an appropriate external standard, concurrently or predictively. Construct evidence tests theoretically expected relationships, differences and structure. Face validity is an appearance judgement and cannot carry the argument alone. Responsiveness adds a longitudinal question about detecting real change.
A familiar measure may be reliable and validated elsewhere yet unsuitable for a new population, language, setting or decision threshold.
Keep diagnostic denominators attached
Sensitivity conditions on reference-positive cases and specificity on reference-negative cases. Predictive values reverse the question to the probability of the condition after a result and therefore change with prevalence.
The ROC curve shows trade-offs across thresholds, while its area summarises discrimination rather than selecting a threshold or proving calibration.
Clinical threshold choice depends on consequences of false negatives and false positives, verification of the reference standard and the pathway in which the test will operate.
Revision drill
For a proposed outcome, state whether the goal is discrimination between people, diagnosis, prediction or evaluation of change. Choose one reliability coefficient, one absolute error quantity and two validity arguments.
Then build a two-by-two diagnostic table and recalculate predictive value after changing prevalence while holding sensitivity and specificity fixed. Explain why the patient-facing interpretation changes even though the test characteristics did not.
What this chapter covers
- 01
Signal, random error and systematic bias
- 02
Test-retest and rater reliability
- 03
Correlation versus agreement
- 04
Face, content, criterion and construct validity
- 05
Sensitivity and specificity
- 06
Predictive values and ROC thresholds
Worked example · free
Reject correlation as proof of agreement
- 4Separate rank consistency from absolute agreement.
- 4Choose bias and agreement analyses.
Key terms
- Reliability
- Consistency of measurement under specified occasions, raters or forms.
- Agreement
- Closeness of scores on the original scale rather than similarity in rank.
- Construct Validity
- Evidence that scores relate to variables and groups as theory predicts.
- Sensitivity
- The proportion of reference-positive cases identified by the index test.
- Specificity
- The proportion of reference-negative cases identified by the index test.
Reliability, Validity and Diagnostic Accuracy FAQ
Can a measure be reliable but invalid?
Yes. A device can produce nearly the same biased reading each time, or a coherent item set can measure the wrong construct consistently. Reliability limits possible validity but does not establish content, criterion, construct or decision validity.
Which denominator belongs to sensitivity?
Sensitivity uses people who truly have the condition according to the reference standard: true positives divided by true positives plus false negatives. Specificity uses people without the condition. Writing the table before calculating prevents denominator reversal.
Why do predictive values change across settings?
Positive and negative predictive values depend on the prevalence or pretest probability in the tested population. When the condition is rare, false positives can outnumber true positives even with strong sensitivity and specificity. Transfer requires a comparable setting and spectrum of patients.
Exam move
Make a two-column table separating reliability questions from validity questions. Practise a diagnostic two-by-two table until all four cells and denominators are automatic. For each coefficient, state whether the intended use concerns ranking, absolute agreement, classification or detection of change.
Working through Reliability, Validity and Diagnostic Accuracy in RS517? Sia is AskSia’s AI Statistics tutor — ask any RS517 Reliability, Validity and Diagnostic Accuracy question and get a clear, step-by-step explanation grounded in how RS517 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.