BMS5010 Chap.5 Reading a Confusion Matrix Without Being Fooled
Reading a Confusion Matrix Without Being Fooled
Four counts, and the four totals built from them
For a binary decision the course summarises performance in a two by two table, described as how well a model separates patients who have the condition from those who do not.
True positive, true negative, false positive and false negative are defined in terms of what happened to a person rather than in terms of arithmetic, and the four sums are written out explicitly: the patients who have the condition, the patients who do not, everyone the model called positive, and everyone it called negative. The first two totals are facts about the patients; the last two are facts about the model.
Every ratio in the topic divides by one of those four, and which one it divides by is the entire meaning of the ratio.
Row ratios describe the test
Sensitivity, also called recall or the true positive rate, is the chance that a patient with the condition receives a positive result. Specificity, the true negative rate, is the chance that a patient without it receives a negative one.
Both are computed inside a group defined by the patient rather than by the model, so moving the same test to a population where the condition is rarer leaves them unchanged. That stability is why manufacturers quote them and why they are the wrong numbers to give a patient.
They are also not independent: lowering the decision threshold raises sensitivity and lowers specificity, so either figure quoted alone is uninterpretable without the other at the same operating point.
Column ratios describe a result
Precision, the positive predictive value, is the share of flagged cases that were genuine.
The negative predictive value is the share of cleared cases that were genuinely clear, and the course notes that this is the figure that matters most in screening, where nearly every result is negative and the guarantee worth having is that a negative really is negative. Both divide by a total the model produced, so both move when the underlying prevalence changes even though nothing about the test has changed.
Confusing a row ratio with a column ratio is the most frequent error in clinical writing about model performance, and the fix is to name the denominator before saying the number.
Why the overall proportion correct is listed first and trusted last
Accuracy takes the whole table as its denominator, and the course states its limitation immediately: it can mislead on imbalanced datasets.
The mechanism is easy to state and easy to check. When one class is rare, a model that always answers with the common class scores close to perfect while identifying none of the cases anybody cares about. It has learned the base rate rather than the condition.
The check costs nothing: work out what the majority-class answer would have scored on your own test set, which is simply the share of that class, and see whether your model clearly beats it.
The balanced summary, and what it cannot tell you
The course gives one combined figure, a harmonic mean combining precision with recall, which stays low unless both are high and so cannot be inflated by pushing one to its ceiling.
It is useful when the two need balancing and it is not a substitute for reporting them: the harmonic mean of ninety and ten is about eighteen, which tells a reader something is badly wrong without saying which of the two it is. Report the pair, then the summary.
Which of the two matters more is decided by what the positive result triggers, and that is a clinical decision made before the model is evaluated rather than a reporting choice made after.
Sweeping the threshold instead of fixing it
Every ratio so far assumes a cut-off has already been chosen, but most models produce a score rather than a decision, so the cut-off is free and moving it trades the two error types.
The curve the course uses plots the true positive rate against the false positive rate, which is one minus specificity, as the cut-off sweeps its range, and the area beneath it summarises the sweep: one is perfect, one half is random, and below one half is worse than random.
One model has better discriminating power than another when its curve lies above the other at every level, and the qualifier matters, because two curves that cross describe models that each win in part of the range.
What this chapter covers
- 01
The four cells defined by what happened to a person
- 02
The four totals, and why the denominator is the meaning
- 03
Sensitivity and specificity as properties of the test
- 04
Precision and negative predictive value as properties of a result
- 05
The majority-class baseline that accuracy has to beat
- 06
The combined figure built from precision and recall, and its blind spot
- 07
The threshold sweep, the area beneath it, and crossing curves
Tell a member of staff what their positive result means
- 2Convert the prevalence into counts of carriers and non-carriers.
- 4Fill all four cells from the two published rates.
- 4Compute both predictive values and name the figure that misleads here.
Key terms
- False Negative
- A negative prediction for a patient who has the condition, which leaves that patient missed and reassured at the same time.
- False Positive
- A positive prediction for a patient who does not have the condition, which triggers whatever follows a positive result for somebody who did not need it.
- True Positive Rate
- Another name for sensitivity and recall, being the share of patients with the condition that the model flags.
- F1 Score
- A single figure combining precision with recall through their harmonic mean, which stays low unless both are high and so resists being inflated by one of them alone.
- Operating Point
- The particular decision threshold at which a model's reported sensitivity and specificity were measured, without which neither figure is interpretable.
- Area Under Curve
- A summary of a model's performance across every decision threshold, where one is perfect discrimination, one half is chance and below one half is worse than chance.
Reading a Confusion Matrix Without Being Fooled FAQ
Which single number should I put in the abstract of a report?
None, if you can avoid it, and if you cannot then the pair that matches the decision. An abstract quoting one figure invites the reader to assume it is the relevant one, and for a rare condition that assumption is usually wrong. The defensible short form names the operating point and gives two numbers: at a threshold flagging one patient in six, the model reached this precision and this recall.
That is barely longer than a single figure and it cannot be misread, which is the whole purpose of an abstract sentence.
Why does a model validated in a clinic disappoint in general practice?
Usually for arithmetic reasons rather than because the model degraded. A specialist clinic sees a much higher proportion of genuine cases than a general practice does, and every ratio dividing by a column total moves with that proportion. The same test with the same sensitivity and specificity therefore reports a much weaker positive predictive value in the wider setting.
This is why a report that quotes a predictive value without stating the prevalence of its own cohort has published a figure nobody can transfer anywhere.
How do I compare two models whose curves cross?
By deciding first which part of the range you will operate in, then reading the curves there. The area beneath each curve averages over every threshold including ones you will never use, so two crossing curves cannot be ranked honestly by their areas.
A screening programme with little tolerance for false alarms lives in the low false-positive region and should choose whichever curve is higher there, even if its overall area is slightly smaller. Say in the report where you operate and why, and the comparison becomes uncontroversial.
Assessment move
Draw the empty two by two table from memory, then write each of the five ratios next to the cells it divides by rather than as a formula. Do it twice, once naming the row denominators and once the column ones. After that, take any published performance figure you can find and say out loud which denominator it used before reading how the authors interpreted it.
Working through Reading a Confusion Matrix Without Being Fooled in BMS5010? Sia is AskSia’s AI Artificial Intelligence tutor — ask any BMS5010 Reading a Confusion Matrix Without Being Fooled question and get a clear, step-by-step explanation grounded in how BMS5010 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
