DATA2002 Chap.14 Model selection and assessing performance
Model selection and assessing performance
The previous chapter ended on an uncomfortable fact: adding a predictor can never increase the residual sum of squares, so it can never reduce r-squared. Any procedure that chooses a model by how well it fits the data it was built from will therefore choose the largest model available every time, regardless of whether the extra terms mean anything.
Over fitting is what that produces, and it is worth defining by what it does rather than by how it feels: a model over fits when it has enough freedom to follow features of the particular sample that will not recur in a new one. Nothing in the fitted output announces it, which is why it must be detected by an outside measurement. The unit teaches two instruments.
An information criterion keeps the in sample fit but attaches an explicit price per parameter, and both versions are penalised measures of badness, so lower is better and a difference of less than one or two units should not be treated as a decision. A model selected by such a criterion can retain a predictor whose own p-value is well above the usual threshold, because the criterion and the t test ask different questions.
Cross-validation abandons in sample measurement altogether: split the observations into folds, hold each out in turn, fit on the rest and predict the held out fold, then pool the errors. Every observation is predicted exactly once by a model that never saw it, and none is wasted.
One rule governs its use and is the most common way it is misapplied: every step that used the response variable, including screening predictors and running a search, must happen inside each fold rather than once beforehand.
What this chapter covers
- 01
Why in sample fit always improves, as arithmetic rather than as evidence
- 02
Over fitting defined by what it does, and why it is invisible in the output
- 03
The shape of an information criterion: reward for fit, price per parameter
- 04
Lower is better, and how large a difference has to be to mean anything
- 05
Why a selected model can retain a term that is not individually significant
- 06
Forward selection, backward elimination and exhaustive search
- 07
Why the two stepwise directions need not agree, and what a disagreement tells you
- 08
Selection optimism, over and above in sample optimism
- 09
The fold structure, and why every observation is predicted exactly once
- 10
How many folds, and the trade between computation and representativeness
- 11
What must be inside the loop, and the mistake of cross-validating only the winner
- 12
What cross-validation estimates: the performance of a procedure, not of one fitted model
Explain why a reported figure is optimistic, twice over
- +1.5First, r-squared is computed on the same sixty observations the model was fitted to, so it measures fit rather than prediction, and it can only rise as predictors are added.
- +1.5Second, and separately, the model was selected by searching many candidates on those same observations, so the reported figure is the best of many noisy quantities and is optimistic for that reason as well.
- +1.5An honest figure comes from cross-validation with the entire procedure inside the loop: within each fold, run the backward search on the training part only, then evaluate the resulting model on the held out part.
- +0.5Running the search once on all the data and then cross-validating only the winning model leaves the selection optimism completely untouched, while producing a figure that looks rigorous.
Key terms
- Over fitting
- Fitting features of the particular sample that will not recur in a new one. It depends on the amount of freedom relative to the amount of data rather than on the number of predictors in the abstract.
- In sample error
- Prediction error measured on the same observations the model was fitted to. It falls monotonically as complexity grows and therefore cannot select a model.
- Out of sample error
- Prediction error measured on observations the model did not see. It has a minimum at some level of complexity, which is what makes it usable for selection.
- Optimism
- The amount by which the in sample error understates the true prediction error. It grows with model complexity, because a model with more freedom can bend further towards its own fitting data.
- Information criterion
- A measure combining fit with an explicit penalty per parameter. Lower values indicate the preferred model, and one common version penalises more heavily than the other once the sample is moderately large.
- Forward selection
- A search that starts from no predictors and adds the one improving the criterion most at each step, stopping when no addition improves it.
- Backward elimination
- A search that starts from every candidate predictor and removes one at a time, which cannot begin when there are more candidates than observations.
- Exhaustive search
- Evaluating every subset of the candidate predictors. It finds the best subset for the criterion and is only feasible for a modest number of candidates.
- Selection optimism
- The additional optimism created by choosing the reported model from among many, since the winner is the maximum of several noisy quantities.
- Fold
- One of the equally sized parts the observations are split into for cross-validation. Each fold is held out in turn while the model is fitted on the remainder.
Model selection and assessing performance FAQ
Why can r-squared never fall when I add a predictor?
Because the model without the new predictor remains available as a special case: setting its coefficient to zero recovers the previous fit exactly. Least squares will therefore never do worse than that, so the residual sum of squares can only stay the same or fall, and r-squared can only stay the same or rise.
A criterion that always rewards more complexity cannot be used to decide how much complexity to have, which is the whole reason this chapter exists.
What counts as a meaningful difference in an information criterion?
More than one or two units. The unit is explicit that models within that band can be regarded as fitting about equally well, so declaring a winner on a gap of half a unit is over reading the criterion. When two candidates sit within the band, choose on other grounds such as interpretability or the cost of measuring an extra predictor, and say that is what you did rather than reporting a false winner.
Should I worry if a selected model contains a predictor with a large p-value?
No, and it is not an inconsistency to be tidied away. The criterion weighs the improvement in overall fit against a fixed price per parameter; a t test asks whether one coefficient is distinguishable from zero given the rest of the model. Those are different questions and they can reach different answers.
Removing terms after selection because their individual p-values are large amounts to running a second, undeclared selection procedure.
Why might a forward and a backward search disagree?
Because both are greedy: each takes the locally best step and never reconsiders. They explore different parts of the space and there is no guarantee they end at the same model. A predictor useful only alongside another may never be added by a forward search, because it does not help on its own.
When the two disagree, that is information about how flat the criterion surface is rather than a bug to be resolved by picking one and not mentioning the other.
How many folds should I use?
The unit's working range is between five and ten. More folds means each fit sees more of the data, so each fitted model is closer to the one you would actually deploy, at the cost of fitting the model more times. Fewer folds means a larger held out set each time, which is faster but risks a test set that is not representative. Whichever you choose, report it, because the estimate depends on it.
Exam move
Hold on to one picture: two curves against complexity, one falling forever and one with a minimum. Everything in this chapter is a comment on that picture, and being able to draw it and say which curve is which is most of the conceptual content.
Second, memorise the direction of the criterion and the size of a meaningful difference together, since a question typically supplies two values and asks what to conclude, and both facts are needed.
Third, practise saying what cross-validation estimates: the performance of a procedure applied to a dataset of this size, rather than of one particular fitted model, which is why the fold models are thrown away and the reported model is refitted on everything.
Fourth, learn the inside the loop rule as a specific claim rather than a slogan, and be able to name what belongs inside: screening predictors by their relationship with the response, running a search, and choosing a tuning constant. Fifth, note that the three quantities conventionally written with the letter k in this unit are unrelated, and name the quantity rather than the letter when you write.
Working through Model selection and assessing performance in DATA2002? Sia is AskSia’s AI Statistics tutor — ask any DATA2002 Model selection and assessing performance question and get a clear, step-by-step explanation grounded in how DATA2002 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
