ECON7880 Chap.7 Overfitting, Validation and Regularisation
Overfitting, Validation and Regularisation
Choose complexity on unseen folds
Training performance improves with flexibility and therefore cannot select model complexity honestly. This chapter uses k-fold cross-validation to estimate mean generalisation and stability, while preserving one untouched test set for the chosen procedure.
Penalise complexity inside the objective
L2 shrinks squared coefficients and L1 can set coefficients to zero.
Grid search evaluates penalty strengths inside training data, and nested validation separates tuning from an outer estimate when procedures themselves must be compared.
Give training, validation and test distinct jobs
Training data fit coefficients. Validation data compare features, hyperparameters and procedures. The untouched test set evaluates the selected pipeline once.
A test set is spent when its outcomes influence any choice, even when no coefficient is fitted on those rows. Repeated test checking turns the holdout into another validation surface and makes its best score optimistic.
Build folds that imitate deployment
Shuffled k-fold validation assumes observations are exchangeable.
Time-ordered prediction needs later holdouts, repeated entities need grouped folds and transfer between locations may need site-level partitions. Every learned preparation step belongs inside the folds.
Imputation, scaling or feature selection performed once before validation gives held-out rows influence over the fitted pipeline.
Read mean performance beside stability
The mean cross-validation score estimates average generalisation, while its spread shows sensitivity to the partition. A tiny mean advantage with much larger variation is a near tie, not decisive superiority.
Inspect unusual folds for shifts in time, group mix or target prevalence. A simple baseline must be processed through the same folds so added complexity earns its place on comparable evidence.
Keep parameter conventions explicit
A penalty multiplier such as ridge alpha strengthens regularisation when it increases.
The C parameter used for logistic regression and SVM moves in the opposite direction: larger C means weaker regularisation. Write the objective and parameter name before stating a direction. Scale predictors inside training data so coefficient penalties reflect modelling choices rather than arbitrary measurement units.
What this chapter covers
- 01
Training and generalisation error
- 02
K-fold cross-validation
- 03
Mean and standard deviation
- 04
Grid search
- 05
Untouched test sets
- 06
L1 and L2 penalties
Worked example · free
Trace a grid-search workflow
- 3Count candidate-fold fits.
- 2Add the training refit.
- 3Protect the final test role.
Key terms
- Generalisation Error
- Prediction error expected on new cases drawn from the deployment setting rather than the training sample.
- Grid Search
- A systematic comparison of specified hyperparameter combinations using a validation procedure.
- Nested Validation
- An outer resampling loop evaluating choices made entirely inside inner tuning loops.
Overfitting, Validation and Regularisation FAQ
What does cross-validation standard deviation show?
It shows how much performance changes across held-out folds. A model with a slightly higher mean but much larger spread may be less dependable on new data from the deployment setting.
When is the test set no longer untouched?
It is spent as soon as its outcomes influence feature selection, model choice, hyperparameters or any other modelling decision, even if no coefficient was fitted on it.
How do L1 and L2 differ?
L2 penalises squared coefficients and tends to shrink them smoothly. L1 penalises absolute coefficients and can drive some exactly to zero, creating automatic feature selection within the fitted model. Compare penalty strengths inside validation after scaling predictors on training data. The test set should not select between them or their feature sets.
Exam move
Draw the split between training, validation and test data, then place each modelling decision on the correct side. Count candidate fits in a small grid search and identify the final training refit. Redesign ordinary folds for a repeated-customer table and a time-ordered table. Rehearse alpha and C directions separately so parameter conventions are not reversed.
Compare two nearly tied models using mean, variation, complexity and baseline performance, then state why checking the test set again after changing the model would invalidate the clean estimate. Sketch nested validation with one outer fold held aside while tuning occurs only in the inner folds. Add imputation and scaling inside that diagram.
Practise reporting a disappointing final holdout without returning repeatedly to tune against it, and name the new data needed for another clean estimate. Add a baseline to every fold and state the smallest improvement that would matter operationally before comparing candidates.
Working through Overfitting, Validation and Regularisation in ECON7880? Sia is AskSia’s AI Economics tutor — ask any ECON7880 Overfitting, Validation and Regularisation question and get a clear, step-by-step explanation grounded in how ECON7880 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
