ELEC5622 Chap.11 Evaluating Classifiers: Cross-Validation, ROC and McNemar
Evaluating Classifiers: Cross-Validation, ROC and McNemar
This chapter is about judging classifiers honestly. Cross-validation estimates performance on unseen data, with hand-out, leave-one-out and V-fold variants. For continuous scores, the ROC curve traces sensitivity against the false positive rate over all thresholds, summarised by AUC. McNemar's test then asks whether two classifiers' error rates really differ on the same cases, counting only the cases where the two disagree.
Small medical data sets make these choices matter, because a few subjects can swing an accuracy estimate. Worked numbers cover a confusion table, an ROC curve from ten scores and a McNemar comparison.
What this chapter covers
- 01
Discriminability on unseen data
- 02
Hand-out validation and its pessimistic bias
- 03
Leave-one-out and V-fold cross-validation
- 04
Sensitivity, specificity and thresholds
- 05
ROC curves and the area under the curve
- 06
McNemar's test for paired classifiers
Worked example · free
Screening rates from a confusion table
- 1Sensitivity = 36 / (36 + 4) = 0.90.
- 1Specificity = 50 / (50 + 10) = 0.833.
- 1The ROC point is (1 − 0.833, 0.90) = (0.167, 0.90).
Key terms
- Leave-one-out validation
- Leave-one-out validation trains on all samples but one, tests on that one, and repeats for every sample.
- Area under the curve
- The area under the ROC curve summarises ranking ability, from 0.5 for chance to 1 for perfect.
- McNemar's test
- McNemar's test compares two classifiers on the same cases using only the cases where they disagree.
- Specificity
- Specificity is the share of truly negative cases a test correctly classifies as negative, TN divided by TN plus FP.
- Cross-validation
- Cross-validation estimates performance on unseen data by rotating which samples are held out for testing.
Evaluating Classifiers: Cross-Validation, ROC and McNemar FAQ
Why is hand-out validation pessimistically biased?
Part of the data is held back and never used for training, so the classifier learns from fewer samples than it could. Its measured error is therefore usually higher than the error of a classifier trained on all the data.
What does an AUC of 0.5 mean?
It is the diagonal of the ROC plot, the performance of random guessing. The lecture puts the floor for an informative test at 0.5 and the ceiling at 1, so useful classifiers sit between them.
Why does McNemar's test ignore cases both classifiers get right?
Those cases, and the ones both get wrong, carry no information about which classifier is better. Only the disagreements, where one is right and the other wrong, can reveal a difference in error rates.
When is V-fold cross-validation preferable to leave-one-out?
On large data sets. Leave-one-out needs one training run per sample, which becomes expensive, while V-fold needs only v runs and still validates every sample exactly once, at the cost of some randomness in how folds are drawn.
Does a higher AUC always mean a better classifier?
A higher AUC means better ranking across all thresholds, and if one ROC curve dominates another its AUC is larger. Two curves can cross, though, so a classifier with lower AUC may still perform better at the threshold a clinic will use.
How do you count McNemar's statistic?
Count n01, the cases only classifier A gets wrong, and n10, the cases only B gets wrong. Then z equals the absolute difference minus one, divided by the square root of their sum. Compare z squared with 3.84, the chi-square value for one degree of freedom at 0.05.
Exam move
Build a small confusion table and compute sensitivity and specificity until it is quick, then plot ROC points for several thresholds of an invented score list and count correctly ranked pairs for the AUC. Learn McNemar's statistic with its continuity correction and the 3.84 cut-off, and practise one example that is significant and one that is not.
Write a one-paragraph evaluation plan for Project 1 naming the validation scheme, the metric and the test you would use to compare two classifiers. Practise reading an ROC plot aloud: what each axis means, where chance lies, and what a step up or a step right says about the next case.
Working through Evaluating Classifiers: Cross-Validation, ROC and McNemar in ELEC5622? Sia is AskSia’s AI Electrical Engineering tutor — ask any ELEC5622 Evaluating Classifiers: Cross-Validation, ROC and McNemar question and get a clear, step-by-step explanation grounded in how ELEC5622 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
