The Hong Kong Polytechnic University · FACULTY OF STATISTICS

RS517 Chap.9 Reliability, Validity and Diagnostic Accuracy

- one subject, every graph, every model, every mark
6 Chapters4-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 9 of 10 · RS517

Reliability, Validity and Diagnostic Accuracy

Consistency is not correctness

Reliability concerns consistency across occasions, raters or forms. Validity concerns whether evidence supports the intended interpretation and use.

A stable biased measure can be highly reliable and invalid for its purpose.

Choose the coefficient from the use

Test-retest, intra-rater, inter-rater, alternate-form and internal-consistency questions require different designs. Correlation shows association but not absolute agreement.

ICC form, kappa, standard error of measurement and agreement plots should be chosen from whether ranking, interchangeability or individual change matters.

Diagnostic denominators

Sensitivity conditions on people with the reference condition and specificity on those without it. Predictive values condition on the test result and change with prevalence.

ROC curves show the sensitivity and false-positive trade-off as a threshold moves.

Choose reliability for the intended decision

Reliability is consistency under specified conditions, so the form must match the use. Test-retest concerns stability across time, inter-rater and intra-rater forms concern scoring conditions, and internal consistency concerns relationships among items.

Correlation alone can remain high when one rater is systematically higher. Name whether consistency or absolute agreement is required, whether a single rating or average will be used, and the population in which the coefficient was estimated.

Accumulate validity evidence instead of seeking one badge

Content evidence asks whether the construct domain is represented.

Criterion evidence compares with an appropriate external standard, concurrently or predictively. Construct evidence tests theoretically expected relationships, differences and structure. Face validity is an appearance judgement and cannot carry the argument alone. Responsiveness adds a longitudinal question about detecting real change.

A familiar measure may be reliable and validated elsewhere yet unsuitable for a new population, language, setting or decision threshold.

Keep diagnostic denominators attached

Sensitivity conditions on reference-positive cases and specificity on reference-negative cases. Predictive values reverse the question to the probability of the condition after a result and therefore change with prevalence.

The ROC curve shows trade-offs across thresholds, while its area summarises discrimination rather than selecting a threshold or proving calibration.

Clinical threshold choice depends on consequences of false negatives and false positives, verification of the reference standard and the pathway in which the test will operate.

Revision drill

For a proposed outcome, state whether the goal is discrimination between people, diagnosis, prediction or evaluation of change. Choose one reliability coefficient, one absolute error quantity and two validity arguments.

Then build a two-by-two diagnostic table and recalculate predictive value after changing prevalence while holding sensitivity and specificity fixed. Explain why the patient-facing interpretation changes even though the test characteristics did not.

In this chapter

What this chapter covers

  • 01

    Signal, random error and systematic bias

  • 02

    Test-retest and rater reliability

  • 03

    Correlation versus agreement

  • 04

    Face, content, criterion and construct validity

  • 05

    Sensitivity and specificity

  • 06

    Predictive values and ROC thresholds

Worked example · free

Reject correlation as proof of agreement

Q [8 marks]. Practice note: the marks used here are not part of the university's published assessment scheme. Two assessors have correlation 0.96, but one scores five points higher. Judge interchangeability.
The diagnostic table fixes denominators: sensitivity conditions on the condition being present, while specificity conditions on it being absent.The diagnostic table fixes denominators: sensitivity conditions on the condition being present, while specificity conditions on it being absent.Reference conditiontrue positivefalse positivefalse negativetrue negativesensitivity uses diseased column; specificity uses non-diseased column
The diagnostic table fixes denominators: sensitivity conditions on the condition being present, while specificity conditions on it being absent.
  • 4Separate rank consistency from absolute agreement.
  • 4Choose bias and agreement analyses.
The assessors rank cases similarly but do not agree on absolute scores. Examine paired differences, mean bias and limits of agreement, and use an ICC form targeting absolute agreement if interchangeability is required. Retraining or calibration may be needed even though correlation is high.
Sia tip — Ask whether ranking or interchangeable scores are needed before selecting the reliability coefficient.
Glossary

Key terms

Reliability
Consistency of measurement under specified occasions, raters or forms.
Agreement
Closeness of scores on the original scale rather than similarity in rank.
Construct Validity
Evidence that scores relate to variables and groups as theory predicts.
Sensitivity
The proportion of reference-positive cases identified by the index test.
Specificity
The proportion of reference-negative cases identified by the index test.
FAQ

Reliability, Validity and Diagnostic Accuracy FAQ

Can a measure be reliable but invalid?

Yes. A device can produce nearly the same biased reading each time, or a coherent item set can measure the wrong construct consistently. Reliability limits possible validity but does not establish content, criterion, construct or decision validity.

Which denominator belongs to sensitivity?

Sensitivity uses people who truly have the condition according to the reference standard: true positives divided by true positives plus false negatives. Specificity uses people without the condition. Writing the table before calculating prevents denominator reversal.

Why do predictive values change across settings?

Positive and negative predictive values depend on the prevalence or pretest probability in the tested population. When the condition is rare, false positives can outnumber true positives even with strong sensitivity and specificity. Transfer requires a comparable setting and spectrum of patients.

Study strategy

Exam move

Make a two-column table separating reliability questions from validity questions. Practise a diagnostic two-by-two table until all four cells and denominators are automatic. For each coefficient, state whether the intended use concerns ranking, absolute agreement, classification or detection of change.

Working through Reliability, Validity and Diagnostic Accuracy in RS517? Sia is AskSia’s AI Statistics tutor — ask any RS517 Reliability, Validity and Diagnostic Accuracy question and get a clear, step-by-step explanation grounded in how RS517 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 24 of your subjects at The Hong Kong Polytechnic University, plus 1,000+ more Bibles across the AskSia library.
Sia - your RS517 tutor, unlimited, worked the way the exam marks it
The full 4-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works