MATH7005 Chap.4 Gradient Descent and Adaptive Optimisers
Gradient Descent and Adaptive Optimisers
Training as repeated local movement
Gradient descent computes the current loss slope and moves parameters in the opposite direction. The chapter records current parameters, fresh gradients, update amounts and new losses at every iteration.
Momentum, RMSProp and ADAM are then compared through the state each retains and the coordinates each rescales.
What to verify
A decreasing training loss proves only that the immediate objective is being reduced.
Validation behaviour, fixed comparison conditions and layerwise gradient evidence are needed before claiming that an optimiser or learning rate improved the useful model.
Geometry comes before brand names
Badly scaled features can make the loss surface narrow and tilted, causing a fixed-step method to zigzag. Standardisation may improve the path before an adaptive method is introduced.
Plot or inspect the update trajectory on a simplified objective so learning-rate effects are not confused with the memory and rescaling mechanisms of the optimiser.
Comparison requires one controlled budget
Hold the split, preprocessing, architecture, initialisation policy and evaluation metric constant. Record time to a nominated validation level, best validation result and the selected checkpoint.
These measurements distinguish quick progress from a better destination. Repeated seeds reveal whether an apparent improvement is stable or simply one favourable run.
Curves are evidence with limits
Oscillation, divergence, overfitting and a disconnected graph can all produce poor training behaviour for different reasons. Pair the curve with gradient norms and validation evidence before changing settings.
Save the optimiser state when resuming training; loading model weights without its moments creates a different trajectory even if the visible parameters begin at the same values.
What this chapter covers
- 01
Gradient direction
- 02
Learning-rate scale
- 03
Fresh slope each step
- 04
Momentum state
- 05
Adaptive rescaling
- 06
Training and validation curves
Worked example · free
Take two quadratic updates
- 2Differentiate the loss.
- 3Compute the first update.
- 4Recompute the slope and update again.
Key terms
- Gradient Descent
- An iterative method that subtracts a scaled loss gradient from the parameters.
- Learning Rate
- The multiplier controlling the size of each parameter update.
- Momentum
- A moving gradient direction that accumulates persistent signals across steps.
- RMSProp
- An adaptive method that scales updates using recent squared gradients.
- ADAM
- An optimiser combining moving first and second gradient moments with bias correction.
Gradient Descent and Adaptive Optimisers FAQ
Why must the gradient be recomputed?
The loss slope depends on the current parameter location. Once an update moves the model, the old derivative describes the previous point and may have the wrong magnitude or even direction.
Does lower training loss mean a better product?
Not necessarily. It shows closer fit to the training objective. Validation performance, operational metrics and failure cases are required to judge whether the learned pattern transfers to records the optimiser did not see.
How should I compare two optimisers?
Hold the data split, preprocessing, initialisation policy and evaluation metric constant. Report convergence speed and final validation performance separately, since a fast route to a weaker solution is not an automatic win.
How does feature scaling affect optimisation?
Large scale differences can create a narrow, tilted loss surface that makes fixed steps zigzag. Standardising training features may improve the geometry before momentum or adaptive rescaling is introduced, and the same transformation must be reused on validation data.
Which checkpoint should the report describe?
Use the checkpoint that produced the reported validation result and name its selection rule. The last training state may fit the training data more closely while generalising worse, so preserve best and final states separately.
What should an early-stopping record contain?
Name the monitored validation quantity, patience, minimum improvement, maximum budget and restored checkpoint. Report variation across seeds when the stopping epoch changes materially, because one noisy validation sequence can select an unrepresentative state.
Why is a learning-rate comparison easy to misread?
Different rates may reach their best validation result at different epochs, so comparing only one shared checkpoint can favour the wrong setting. Keep a common compute budget, record each trajectory and report the selection rule with the chosen state.
Assessment move
Work a quadratic objective for several steps, recording the parameter, fresh gradient, update and new loss on separate lines. Repeat with a step size that oscillates so the role of learning rate becomes visible. Add a momentum state using one declared convention and compare it with the current gradient.
In a small code experiment, standardise features, hold the data split and initialisation fixed, and compare plain descent with one adaptive method under the same epoch budget. Plot training and validation losses together, log gradient norms and save the checkpoint producing the reported metric.
For the project, keep the optimiser name, convention, learning rate, batch size, stopping rule, seed policy and selected checkpoint in the evidence index.
Working through Gradient Descent and Adaptive Optimisers in MATH7005? Sia is AskSia’s AI Computer Science tutor — ask any MATH7005 Gradient Descent and Adaptive Optimisers question and get a clear, step-by-step explanation grounded in how MATH7005 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
