ECE5179 Chap.6 Overfitting and Regularisation
Overfitting and Regularisation
This chapter gathers the unit's tools for models that fit their training data too well. It shows how training and validation curves reveal overfitting, why the test set must be kept for the final estimate, and how dropout, weight decay and other penalties restrain a model.
It then adds the practical defences from the PyTorch lessons, augmentation and transfer learning with linear probes, and ends with the metrics used to judge a classifier honestly.
What this chapter covers
- 01
Training, validation and test sets; k-fold cross-validation
- 02
Early stopping from validation loss
- 03
Dropout and inverted dropout scaling
- 04
Weight decay, ridge and LASSO penalties
- 05
Augmentation, transfer learning and linear probes
- 06
Accuracy, precision, recall and F1
Worked example · free
Spot overfitting on training and validation curves
- 1Falling training loss with rising validation loss after epoch 13 is the signature of overfitting.
- 1Keep the weights from epoch 13, the minimum of the validation curve, which is early stopping.
- 1Add a regulariser such as dropout or weight decay, tuning its strength on validation data.
- 1Increase effective data through augmentation, or start from a pre-trained network and fine-tune it.
Key terms
- Overfitting
- Overfitting is fitting the training data so closely that performance on new data gets worse.
- Validation set
- A validation set is held-out data used to choose hyperparameters and decide when to stop training.
- Weight decay
- Weight decay shrinks every weight a little on each update by adding a multiple of the weight to its gradient.
- Data augmentation
- Data augmentation trains on label-preserving random transformations of the examples to increase their variety.
- Linear probe
- A linear probe trains only a new final layer on top of frozen features from a pre-trained network.
- Recall
- Recall is the fraction of truly positive cases that a classifier correctly identifies.
Overfitting and Regularisation FAQ
Why must survivors be rescaled in dropout?
If units are zeroed with probability p during training but all fire at test time, the next layer receives more input at test time than it learned to expect. Inverted dropout divides the surviving activations by 1 − p during training, so the expected input is the same in both modes and the test network needs no change.
What is the difference between L2 and L1 regularisation?
An L2 penalty adds the sum of squared weights and shrinks all weights proportionally, which is why it is called weight decay. An L1 penalty adds the sum of absolute weights and tends to push small weights exactly to zero. The elastic net combines them, and the strength of either is tuned on validation data.
Why is accuracy misleading for rare classes?
When positives are rare, a model can be right most of the time simply by predicting the common class, so accuracy looks high while the cases that matter are missed. Precision and recall separate the two kinds of error, and F1 balances them in one number.
How does k-fold cross-validation choose a hyperparameter?
Split the training data into k folds, train k times with each fold held out once, and average the k validation errors for each candidate setting. Pick the setting with the lowest average, then retrain on all the training data. It uses scarce data efficiently and avoids trusting one lucky split.
Should dropout be active when evaluating a model?
No. Dropout is a training-time regulariser, so call model.eval() before validation or testing to switch it off, along with the training behaviour of batch normalisation. With inverted dropout no rescaling is needed at test time, because survivors were already divided by 1 − p during training.
Exam move
Sketch the three classic curves, underfit, good fit and overfit, and say for each what you would change. Practise inverted-dropout arithmetic on a small activation vector and verify the expectation for one unit. Do one weight-decay update by hand in both forms, w − η(∇L + λw) and (1 − ηλ)w − η∇L, and confirm they agree.
Finish with a confusion matrix: compute accuracy, precision, recall and F1, and argue which one a given application should prioritise. Keep a two-column table of regularisers with what each changes, the loss, the weights, the data or the architecture, and the hyperparameter that controls it. Then for each, write one sentence on how you would choose that hyperparameter using validation data, never the test set.
Working through Overfitting and Regularisation in ECE5179? Sia is AskSia’s AI Computer Science tutor — ask any ECE5179 Overfitting and Regularisation question and get a clear, step-by-step explanation grounded in how ECE5179 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
