BMS5010 Chap.6 Overfitting, Validation and Generalisation
Overfitting, Validation and Generalisation
Three states, and none is a property of the model alone
The course frames overfitting, underfitting and good fit as descriptions of how well a model generalises to data it has not seen, and that framing is worth taking literally. Each of the three is a statement about the gap between two measurements, so a model examined on a single dataset cannot be placed in any of them.
Overfitting is capturing noise instead of the underlying pattern, recognised by strong training performance and weak performance on unseen data. Underfitting is being too simple to capture the patterns at all, recognised by weak performance on both.
Good fit is defined negatively, as the absence of both, which means it is a band rather than a point.
Causes and symptoms are different lists
The causes of overfitting are a model with too much freedom, too many parameters, or too few training rows, all versions of too much freedom for too little evidence.
Its symptoms are high training accuracy with low test accuracy, a large gap between training and validation error, and poor performance on unseen data.
The causes of underfitting are too few features, a model with too little freedom, or a penalty set too high, and that last cause matters because regularisation was sold earlier as a defence against overfitting: pushed far enough, the same penalty shrinks the coefficients until the model says nothing.
You diagnose from symptoms and repair from causes, and mixing the two lists is how repair rounds go in circles.
The same triple in every kind of model
In regression, underfitting is a straight line on a curved relationship, good fit is a low-degree polynomial that follows the trend without chasing outliers, and overfitting is a high-degree curve that fits the noise exactly and oscillates between the points it was given.
In classification the equivalents are a linear boundary where the classes are not linearly separable, a modest curved boundary, and a boundary so intricate that it wraps around individual training points.
The clustering version is the one students miss: too few clusters merges distinct groups, too many forces each point toward its own, and the right choice reflects real structure such as distinct biological subpopulations.
Three portions of data, three different jobs
The training portion, usually three fifths to four fifths, is what the model learns from, adjusting its parameters by an optimisation procedure such as gradient descent with the goal of minimising the loss.
The validation portion, usually a tenth to a fifth, tunes settings that are not learned from the data, such as a learning rate or the number of layers, and is used in techniques such as fold-based resampling to get more stable estimates. The test portion, also a tenth to a fifth, is kept completely separate and gives the final unbiased estimate.
The middle portion cannot double as the last one, because every comparison you make on it consumes a little of its independence.
Early stopping, and where to stop
The rule the course gives is direct: when validation error turns upward while training error is still falling, the model has begun to overfit and training should end there.
The model to select corresponds to the point where the validation loss begins to rise, with underfitting before that point and overfitting after it. That makes the turn the only defensible stopping place and makes running for a fixed number of iterations an arbitrary choice presented as a setting.
It also explains why the validation portion earns its cost: training error keeps falling long after memorisation has started, and only a second curve shows where that began.
Leakage, resampling and the validation that actually transfers
The central rule for the test portion is that the model must never see it during training, and the leak usually happens before the split in a step nobody counts as training: standardising with statistics from everything, selecting variables by examining everything, or filling missing values from everything.
Fold-based cross-validation trains on all but one fold and tests on the remainder, repeating so each fold is tested once, and its limit case trains on every sample but one. The course names the price of that limit case as extreme slowness and high variance.
Internal validation checks how well a model fits the data you have; only an independent cohort collected at another hospital, from another population, or in another period answers whether it transfers.
What this chapter covers
- 01
Three states, each defined as a gap between two measurements
- 02
Causes against symptoms, and why the two lists must not be mixed
- 03
The same triple in regression, in classification and in clustering
- 04
Training, validation and test portions and the job each one does
- 05
Early stopping at the turn in the validation curve
- 06
Leakage, and the order of operations that prevents it
- 07
Fold-based resampling, its limit case, and its stated costs
- 08
Internal validation against an independent cohort
Find the leak in a procedure that reports a strong result
- 3Name the two steps that used the held-back data.
- 3Explain what each one transferred across the split.
- 3State the corrected order of operations and what to expect from it.
Key terms
- Generalisation
- How well a model performs on data it did not learn from, which is the only performance question a report can honestly answer.
- Validation Portion
- The share of data reserved for tuning settings that are not learned during fitting, which is consumed by the choices made on it.
- Early Stopping
- Ending training at the point where validation error begins to rise while training error is still falling, which is the only defensible stopping rule.
- Cross-Validation
- Repeatedly splitting one dataset so that every observation is tested on exactly once, giving both an average estimate and a spread around it.
- Internal Validation
- Assessing a model within the dataset it was developed on, which answers how well it fits the available data and not whether it transfers.
Overfitting, Validation and Generalisation FAQ
Why report the spread across folds and not just the average?
Because the spread is what tells a reader whether the average means anything. Ten folds that agree closely say the estimate is stable and would survive a different random split; ten folds ranging widely say the dataset is too small or too varied for the headline figure to be relied on, and that is a finding worth publishing rather than hiding.
The spread also protects you in the demonstration, because it is the honest answer to a question about how confident you are, and it is far stronger than an opinion.
My training and validation scores are both poor. Is that overfitting?
No, and the difference matters because the repairs are opposite. Poor performance on both portions with a small gap between them is the published symptom of underfitting, which means the model is too simple, has too few features, or has been penalised too hard. The repair is to relax the penalty, add features, or allow a more flexible model, then watch the gap reopen and stop at the turn.
Treating it as overfitting and simplifying further makes it worse, which is why the symptom list is worth learning before the cause list.
Can cross-validation replace an external cohort?
No. Every fold is drawn from the same dataset, so resampling improves the stability of an internal estimate without addressing anything that is uniformly true of that dataset, such as one hospital's recording habits, one instrument's calibration or one period's case mix.
The course maps external validation onto the test phase and describes it as an independent dataset collected at another hospital, from another population, or in another period. If you have a second cohort, keep it aside entirely rather than pooling it, because pooling converts your only transfer evidence into a slightly larger internal validation.
Assessment move
Take any notebook you have written and list every operation in order, then mark each one as learns from data or does not. Draw the split line where it currently falls and see how many marked operations sit above it. That exercise finds leaks faster than reading about them, and the resulting list is the methods paragraph a reviewer wants.
Working through Overfitting, Validation and Generalisation in BMS5010? Sia is AskSia’s AI Artificial Intelligence tutor — ask any BMS5010 Overfitting, Validation and Generalisation question and get a clear, step-by-step explanation grounded in how BMS5010 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
