ETC1000 Chap.9 Using Models for Classification and Prediction
Using Models for Classification and Prediction
The outcome becomes a category
Every model so far predicted a quantity and measured its errors as residuals in the units of that quantity.
Predicting whether a customer will churn or an invoice will be paid late leaves nothing for a residual to mean, so fit has to be judged by counting correct and incorrect calls instead.
A model gives a probability; a decision needs a threshold
A classifier returns an estimated probability that a case belongs to the positive class, and someone has to choose the cut off above which that becomes a yes.
One half is the obvious choice and it is only correct when the two kinds of mistake cost the same, which in business they rarely do. Moving the threshold does not improve the model; it trades one error for the other.
Four counts, and four different denominators
Crossing actual class against predicted class puts every case in one of four cells. Accuracy divides the two correct cells by everything.
Sensitivity restricts to the cases that really were positive and asks what share was caught. Specificity restricts to the actual negatives. Precision restricts instead to the cases the model called positive.
Sensitivity and precision differ only in whether the denominator is the actual column or the predicted row.
Accuracy is dominated by the base rate
When the positive class is rare, the arithmetic rewards ignoring it. A model that never predicts default on a portfolio with a 3 per cent default rate scores 0.97 accuracy and catches nothing.
Ask what share of the data is in the positive class before reading any accuracy figure.
What this chapter covers
- 01
Why residuals stop being defined once the outcome is a category
- 02
The threshold as a business decision rather than a model output
- 03
The four counts and the four rates built from them
- 04
Sensitivity against precision, and the denominator that separates them
- 05
The base rate, and why accuracy alone flatters a useless model
- 06
Scoring on data the model has not seen
Score a classifier against the benchmark it has to beat
- 2Compute accuracy, sensitivity and precision.
- 1Compare accuracy with the do nothing benchmark.
- 1Say whether the model is useful for a retention campaign.
Key terms
- Classification
- Predicting which of two labels a case belongs to rather than a value on a scale.
- Threshold
- The estimated probability above which a case is called positive.
- Confusion matrix
- The two by two table of actual against predicted class holding the four counts.
- Accuracy
- The share of all cases the model called correctly.
- Sensitivity
- The share of actual positive cases the model caught.
- Specificity
- The share of actual negative cases the model cleared.
- Precision
- The share of the cases called positive that really were positive.
- Base rate
- The share of the data belonging to the positive class, which any model must beat.
Using Models for Classification and Prediction FAQ
Why is accuracy a poor headline number?
Because it is dominated by whichever class is larger. On a portfolio with a 3 per cent default rate, a model that predicts no defaults at all scores 0.97, catches nothing and has no value for the decision it was built for. Accuracy belongs in the report, but never on its own and never before the base rate it has to beat.
What is the difference between sensitivity and precision?
Only the denominator, in exactly the way that marginal and conditional probabilities differ in week 1. Sensitivity divides the true positives by all the cases that actually were positive. Precision divides the same number by all the cases the model called positive. One measures how much of the target was found and the other how trustworthy a flag is.
How do I choose the threshold?
From the relative cost of the two errors, not from convention. If missing a positive case costs many times what a false alarm costs, lower the threshold well below one half: you will catch more genuine cases and make more wasted interventions, which is the correct trade. If acting on a false positive is expensive, raise it.
Why must a model be scored on data it has not seen?
Because the fitting procedure is built to do well on the rows it was given, so scoring on those rows flatters the model by construction. Holding part of the data back, fitting on the rest and scoring the held back part is what distinguishes a model that learned a pattern from one that memorised a sample.
Exam move
Practise reconstructing all four rates from a table of four counts until the denominators are automatic, because that is the same skill week 1 tested on contingency tables. Then rehearse the two sentence report: the rate that matters for the decision, and the base rate it has to beat.
Working through Using Models for Classification and Prediction in ETC1000? Sia is AskSia’s AI Statistics tutor — ask any ETC1000 Using Models for Classification and Prediction question and get a clear, step-by-step explanation grounded in how ETC1000 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
