BME5150 Chap.3 Evaluating Classifiers and Explanations
Evaluating Classifiers and Explanations
Begin with the four classification counts
True positives, false negatives, false positives and true negatives preserve the errors hidden by accuracy. Sensitivity conditions on actual positives, specificity on actual negatives, and precision on positive predictions. Recall is another name for sensitivity.
Writing the denominator in words before calculating prevents the most common confusion.
Accuracy is vulnerable to prevalence
When negative cases dominate, predicting the majority class can create high accuracy while finding few positive cases. Class-specific rates and counts show the problem.
Precision also depends on prevalence, so it can change across settings even when sensitivity and specificity remain similar.
ROC and AUC describe ranking
A ROC curve traces true positive rate against false positive rate as the threshold moves. AUC summarises the probability that a randomly chosen positive case receives a higher score than a randomly chosen negative case.
It does not select a deployment threshold or state the probability that one prediction is correct.
Threshold choice is a clinical decision
A looser threshold often finds more positive cases and creates more false alarms. A stricter threshold does the reverse. The acceptable operating point depends on the consequence of a missed case, the burden of additional review and the resources available after an alert.
Report the threshold beside sensitivity, specificity and precision.
Explanation is not causation
SHAP values and feature importance describe how a fitted model uses available predictors under a chosen reference or aggregation. Correlation, preprocessing and background data shape those attributions. An important scanner variable may reveal site confounding rather than a disease mechanism.
Use explanation to audit behaviour, then design separate evidence for causal or biological claims.
What this chapter covers
- 01
Confusion-matrix counts and denominators
- 02
Sensitivity, specificity and precision
- 03
Accuracy under class imbalance
- 04
ROC curves and AUC
- 05
Threshold selection and clinical workload
- 06
SHAP and feature-importance boundaries
Worked example · free
Translate classifier rates into clinical counts
- 2Complete TP, FN, FP and TN.
- 4Calculate four metrics from the correct denominators.
- 4Express missed cases and extra reviews as the trade-off.
Key terms
- Confusion Matrix
- A confusion matrix records correct and incorrect predictions for each actual outcome class.
- Sensitivity
- Sensitivity is true positives divided by all actual positive cases.
- Specificity
- Specificity is true negatives divided by all actual negative cases.
- Precision
- Precision is true positives divided by all cases predicted positive.
- ROC Curve
- A ROC curve plots sensitivity against false positive rate across decision thresholds.
- SHAP Value
- A SHAP value attributes part of a fitted prediction to a feature relative to a reference.
Evaluating Classifiers and Explanations FAQ
Why can sensitivity and precision tell different stories?
Sensitivity begins with actual positive cases, while precision begins with positive predictions. A model can detect most positive cases and still produce many false alarms, especially when prevalence is low. Always name the denominator and report the underlying counts.
Does the highest AUC identify the best clinical threshold?
No. AUC averages ranking behaviour across thresholds. Deployment requires a specific operating point chosen from missed-case consequences, false-alarm burden and capacity. Compare models in the clinically relevant region and report threshold-specific errors.
What can a SHAP plot safely establish?
It can show how a fitted model's predictions are attributed to features under the selected reference and data. It cannot prove a biological mechanism or causal effect. Investigate correlation, site confounding and stability before giving an attribution clinical meaning.
Assessment move
Build a blank confusion matrix and fill it from prose until you can calculate every metric without memorised slogans. For sensitivity and specificity, point to actual-class columns; for precision, point to the predicted-positive row. Sketch a ROC curve and mark two thresholds, then describe how moving between them changes both errors.
Next, take one feature-importance result and write two sentences: one safe model-behaviour statement and one causal overclaim to avoid. End each revision session by translating a percentage into expected counts for a realistic cohort size, because clinical consequences are easier to judge as people and reviews than as decimals. Repeat the exercise after changing prevalence while holding sensitivity and specificity fixed.
Watching precision move makes the population dependence concrete and prevents a case-control result from being read as deployment workload.
Working through Evaluating Classifiers and Explanations in BME5150? Sia is AskSia’s AI Computer Science tutor — ask any BME5150 Evaluating Classifiers and Explanations question and get a clear, step-by-step explanation grounded in how BME5150 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
