The Hong Kong University of Science and Technology · FACULTY OF COMPUTER SCIENCE

MAIE5421 Chap.3 Classification Metrics and the Confusion Matrix

- one subject, every graph, every model, every mark
8 Chapters4-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 3 of 10 · MAIE5421

Classification Metrics and the Confusion Matrix

Three shapes of classification problem

Classification learns from input-output pairs whose outputs are categories.

The lecture distinguishes binary problems with exactly two classes, such as deciding whether an email is spam or verifying that a face matches an enrolled identity; multi-class problems, in which each sample belongs to exactly one of more than two classes, such as recognising which person or which object is in an image; and multi-label problems, in which a sample can carry several labels at the same time, such as tagging a document or a photograph with every topic it contains.

The confusion matrix as the source of every score

For a binary problem, each prediction falls into one of four outcomes.

True positives and true negatives are correct. A false positive is a false alarm, a negative sample wrongly flagged; a false negative is a missed detection, a positive sample wrongly passed.

Laying the four counts out as a two-by-two table, with the prediction on one axis and the ground truth on the other, gives the confusion matrix, and every metric in the chapter is a ratio of its cells.

Four metrics and what each protects against

Accuracy is the share of all samples classified correctly, and the lecture warns that it is unreliable on imbalanced datasets, where always predicting the majority class already scores well.

Precision is the share of predicted positives that are real, so it matters when false positives are costly. Recall is the share of real positives that are detected, so it matters when false negatives are costly, as in medical screening.

The F1 score combines precision and recall into one number that is low unless both are reasonably high, which the lecture recommends when the class distribution is uneven.

Beyond two classes

For a multi-class problem the confusion matrix grows to one row and one column per class.

Overall accuracy is still the diagonal divided by the total, and precision, recall and F1 can be computed for each class by treating that class as positive and every other class as negative. For a multi-label problem, a separate two-by-two matrix is built for each label, and each label receives its own accuracy, precision, recall and F1.

In this chapter

What this chapter covers

  • 01

    Binary, multi-class and multi-label classification

  • 02

    True positives, false positives, false negatives and true negatives

  • 03

    Reading the axes of a confusion matrix

  • 04

    Accuracy and why it misleads on imbalanced data

  • 05

    Precision when false alarms are costly

  • 06

    Recall when missed detections are costly

  • 07

    The F1 score as a combined measure

  • 08

    Per-class metrics for multi-class and multi-label problems

Worked example · free

Choose between two pedestrian detectors with the same F1

Q [7 marks]. AskSia-authored practice. On video containing 240 pedestrians, Detector A raises 200 alarms of which 180 are correct, and Detector B raises 300 alarms of which 220 are correct. Compute precision, recall and F1 for both, and recommend one for an emergency-braking system. The marks shown are an AskSia study allocation and are not the University's marking scheme.
Precision, recall and F1 for two pedestrian detectors.Precision, recall and F1 for two pedestrian detectors. Detector A is more precise and Detector B finds more pedestrians, yet their F1 scores differ by only 0.003, so the bars alone cannot choose between them.0.900.73Precision0.750.92Recall0.820.81F1Detector ADetector B
Precision, recall and F1 for two pedestrian detectors. Detector A is more precise and Detector B finds more pedestrians, yet their F1 scores differ by only 0.003, so the bars alone cannot choose between them.
  • 2Detector A: precision 180 / 200 = 0.900, recall 180 / 240 = 0.750.
  • 2Detector B: precision 220 / 300 = 0.733, recall 220 / 240 = 0.917.
  • 2F1: A = 2 × 0.900 × 0.750 / 1.650 = 0.818; B = 2 × 0.733 × 0.917 / 1.650 = 0.815.
  • 1Recommend B for braking, because a missed pedestrian is the costlier error.
The two F1 scores, 0.818 and 0.815, are practically identical, so F1 cannot make the choice. Detector A misses 60 of the 240 pedestrians, while Detector B misses only 20 at the price of 80 false alarms instead of 20. In an emergency-braking system a missed pedestrian can cause injury, whereas a false alarm causes an unnecessary stop, so recall is the metric that reflects the real cost and Detector B is the one to deploy.
Sia tip — Write the cost of a false positive and of a false negative in words before computing anything. The metric to maximise is the one that counts the expensive error.
Glossary

Key terms

Confusion Matrix
A table counting predictions against ground truth, with one cell for each combination of predicted and actual class.
True Positive
A positive sample that the model correctly predicts as positive.
False Positive
A negative sample that the model wrongly predicts as positive, also called a false alarm.
False Negative
A positive sample that the model wrongly predicts as negative, also called a missed detection.
Accuracy
The share of all samples that the model classifies correctly, the diagonal of the confusion matrix over the total.
F1 Score
The harmonic mean of precision and recall, which stays low unless both are reasonably high.
Multi-label Classification
Classification in which one sample can belong to several classes at the same time.
Imbalanced Dataset
A dataset in which one class is much rarer than another, so that accuracy alone flatters weak models.
FAQ

Classification Metrics and the Confusion Matrix FAQ

Why is accuracy misleading on imbalanced data?

Because a model can reach a high accuracy by always predicting the majority class. If 95 of every 100 images are clean, a model that never flags anything is 95% accurate and finds no defects at all. Precision, recall and F1 look at the positive class directly, so they expose a model that ignores it, which is why the lecture prefers them when classes are uneven.

What is the difference between precision and recall?

Precision asks, of everything the model flagged as positive, how much really was positive. Recall asks, of everything that really was positive, how much the model flagged. Precision falls when there are many false alarms and recall falls when there are many misses. Which one to prioritise depends on which error is more expensive in the application.

How do I compute precision for one class in a multi-class problem?

Treat that class as positive and every other class as negative. Its true positives are the diagonal entry for that class. Its false positives are the other samples the model assigned to that class, so precision is the diagonal entry divided by the total number of predictions of that class. Recall divides the same diagonal entry by the number of samples that truly belong to the class.

Study strategy

Assessment move

Draw a blank two-by-two matrix and label the axes before every metric question, then fill the four cells from the wording of the problem. Most errors in this topic come from putting a count in the wrong cell, not from the formulas, and the blank matrix forces you to decide which error is the false alarm and which is the miss before you divide anything.

Working through Classification Metrics and the Confusion Matrix in MAIE5421? Sia is AskSia’s AI Computer Science tutor — ask any MAIE5421 Classification Metrics and the Confusion Matrix question and get a clear, step-by-step explanation grounded in how MAIE5421 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 13 of your subjects at The Hong Kong University of Science and Technology, plus 1,000+ more Bibles across the AskSia library.
Sia - your MAIE5421 tutor, unlimited, worked the way the exam marks it
The full 4-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works