DATA2002 Chap.15 Classification: logistic regression, trees, neighbours
Classification: logistic regression, trees, neighbours
The response is now a category rather than a number, and two things change with it. A straight line model would eventually predict values below zero and above one, which cannot be probabilities, so the linear part is placed on the log-odds scale where it is free to run from minus infinity to plus infinity and is then mapped back.
And the measure of success changes from an error size to a count of correct labels, which turns out to be far easier to misread than a residual standard error. Three scales carry the logistic model and the neutral value moves between them: a coefficient is a log-odds where zero means no effect, its exponential is an odds ratio where one means no effect, and the logistic function gives a probability.
Because the estimates are maximum likelihood estimates whose distribution is normal only in large samples, the printed statistic is compared with a normal distribution rather than a t, and there are no degrees of freedom to quote beside a coefficient. Judging a classifier is where most of the marks sit.
The confusion matrix puts predictions down the rows and truth across the columns, and transposing it leaves the accuracy unchanged while swapping sensitivity and specificity, which is dangerous precisely because nothing looks wrong. Every accuracy figure must be compared with what a model predicting the majority class every time would score, because on an unbalanced outcome that do nothing model can look excellent.
Trees split parallel to the axes and are stopped by a constant that runs the opposite way to intuition, forests average many trees and sample the predictors to decorrelate them, and nearest neighbours has no fitted model at all and one constant that cannot be chosen in sample.
What this chapter covers
- 01
Why a straight line cannot model a zero or one response
- 02
Log-odds, odds and probability, and where the neutral value sits on each
- 03
The three step conversion, and why rounding partway through changes the answer
- 04
Why the printed statistic is a z and carries no degrees of freedom
- 05
The classification threshold as a separate decision from the model
- 06
The confusion matrix, its orientation, and what transposing it silently swaps
- 07
Comparing accuracy with the majority class rate, always
- 08
Accuracy as one minus the resubstitution error, and why it is optimistic
- 09
Trees: axis parallel splits, and the constant that is a required improvement
- 10
Deeper trees against more trees, as two different remedies
- 11
Why a forest samples the predictors at each split
- 12
Nearest neighbours: the degenerate in sample optimum, and the two conditions on the distance
Diagnose a classifier from three summary numbers
- +2It has learned to predict the negative class almost always. A model doing exactly that would score ninety five per cent accuracy, so the reported ninety four per cent is actually worse than doing nothing.
- +1The specificity of 0.99 confirms it rarely flags a negative case, which is what predicting negative almost always produces and is not evidence of skill.
- +1The sensitivity of 0.02 is the informative number: the model catches two per cent of the events it exists to find.
- +1One useful change is to lower the classification threshold below one half, trading specificity for sensitivity, which is appropriate when a missed event costs far more than a false alarm. Reporting the predicted probabilities rather than only the labels would let a user make that trade themselves.
Key terms
- Log-odds
- The logarithm of the odds, which is the scale the linear part of a logistic model lives on. It ranges over all real numbers and its neutral value is zero.
- Odds ratio
- The exponential of a logistic coefficient, describing the factor by which the odds of the outcome are multiplied per unit of the predictor. Its neutral value is one and it can never be negative.
- Logistic function
- The transformation that maps a log-odds back to a probability between zero and one. Applying it is the last step of the conversion chain.
- Classification threshold
- The probability above which the positive class is predicted, conventionally one half. It is a decision separate from the model and can reasonably be moved when the two kinds of mistake have different costs.
- Confusion matrix
- A table of predicted class against true class. Printed output places predictions down the rows and truth across the columns, and transposing it swaps sensitivity and specificity.
- No information rate
- The accuracy a model would achieve by predicting the majority class every time. Every reported accuracy should be quoted against it.
- Resubstitution error
- The error rate measured on the data the model was fitted to. Accuracy is one minus it, and both are optimistic for the same reason as an in sample r-squared.
- Complexity constant
- The minimum improvement a tree split must deliver to be kept. Raising it therefore produces a smaller tree, which is the opposite of the usual intuition.
- Random forest
- An ensemble of trees, each grown on a resample of the observations and offered only a random subset of the predictors at each split. The subsetting decorrelates the trees so that averaging them helps.
- Nearest neighbours
- A classifier that predicts by taking a vote among the closest observations. It has no fitted parameters, so everything depends on the number of neighbours and on how distance is defined.
Classification: logistic regression, trees, neighbours FAQ
Why does the same result get reported on three different scales?
They are three scales for the same information and the neutral value moves between them. A logistic coefficient is a log-odds, neutral at zero, and it may be any real number. Its exponential is an odds ratio, neutral at one, and it is always positive. Applying the logistic function gives a probability between zero and one.
A negative coefficient therefore corresponds to an odds ratio below one, and comparing an odds ratio against zero rather than against one is the most common slip in reading this output.
Why does a logistic model report a z rather than a t?
Because the coefficients are maximum likelihood estimates, and their distribution is normal only in large samples. There is no exact small sample correction to apply and no residual degrees of freedom to quote beside a coefficient, so the printed statistic is compared with a standard normal distribution. Writing degrees of freedom next to a logistic coefficient is a definition error rather than a rounding of the truth.
Why is the confusion matrix orientation such a problem?
Because getting it wrong leaves the accuracy unchanged and swaps sensitivity and specificity, so both remain plausible numbers and nothing on the page looks wrong. Printed output places predictions down the rows and truth across the columns, and the function that builds it takes the predictions first and the truth second.
Passing them the other way round transposes the table silently, which is why the convention is worth memorising rather than deriving on the day.
Why does raising the complexity constant make a tree smaller?
Because the constant is a minimum required improvement rather than a budget for complexity. A split is only kept if it improves the fit by at least that much, so demanding more of each split keeps fewer of them. Reading it as a measure of how complex the tree is allowed to be inverts the direction, and the mistake is invisible until the tree comes out the wrong size.
Why does a forest offer only some predictors at each split?
To decorrelate the trees, not for speed. Without the restriction, a single strong predictor would be chosen at the top of nearly every tree, the trees would be near copies of one another, and averaging near copies gains almost nothing. Offering a random subset forces different trees to use different structure, so their errors differ and averaging cancels part of them. That is the mechanism the ensemble depends on.
Exam move
Almost everything examinable here is a direction, so the direction sheet is the artefact this chapter is revised from and it should exist before anything else does: neutral at zero on the log-odds scale and at one on the odds ratio scale, z rather than t, predictions down and truth across, larger complexity constant giving a smaller tree, more trees rather than deeper ones, predictors sampled to decorrelate, and one neighbour giving perfect in sample accuracy on any dataset.
Each of those is a plausible sounding sentence with a plausible sounding opposite, which is exactly what a multiple choice section is built from. Once the sheet exists, work outward from it in three passes.
The mechanical pass is the three step conversion, computing the linear part, exponentiating to odds, then converting to a probability, practised until it runs without thought and with full precision carried through rather than rounded partway. The reflex pass is computing the do nothing accuracy from any confusion matrix you meet before commenting on the model's, which takes ten seconds and reframes the answer.
The judgement pass is being ready to say when you would move the classification threshold and what you would trade for it, because the threshold is a decision rather than part of the model.
Working through Classification: logistic regression, trees, neighbours in DATA2002? Sia is AskSia’s AI Statistics tutor — ask any DATA2002 Classification: logistic regression, trees, neighbours question and get a clear, step-by-step explanation grounded in how DATA2002 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
