ECE5179 Chap.3 Gradient Descent and Optimisation Geometry
Gradient Descent and Optimisation Geometry
Week 3 explains how large models are trained when no closed-form solution exists. The gradient points uphill and is orthogonal to the level sets, so stepping against it decreases the loss for a small enough learning rate.
The chapter covers the update rule, stopping criteria, critical points and convexity, convergence rates, and how to diagnose a learning rate from a loss curve, ending with the momentum update listed on the formula sheet.
What this chapter covers
- 01
Why grid search fails in high dimensions
- 02
The gradient as the direction of steepest ascent
- 03
Level sets and why paths zigzag in narrow valleys
- 04
The gradient-descent update and stopping criteria
- 05
Critical points, convexity and global minima
- 06
Sublinear and linear convergence
- 07
Learning-rate diagnosis and momentum
Worked example · free
Gradient descent on a quadratic and the best learning rate
- 1Derivative f′(w) = 4w − 8, so the first step gives w = 0 − 0.1 × (−8) = 0.8.
- 1Second step: f′(0.8) = −4.8, so w = 0.8 + 0.48 = 1.28.
- 1Third step: f′(1.28) = −2.88, so w = 1.28 + 0.288 = 1.568.
- 1The minimiser is w* = 8/4 = 2, and each update multiplies the distance w − 2 by 1 − 4η.
- 1One-step convergence needs 1 − 4η = 0, so η = 0.25.
- 1Convergence needs |1 − 4η| < 1, so 0 < η < 0.5.
Key terms
- Level set
- A level set is the collection of points at which a function takes one fixed value.
- Critical point
- A critical point is a point where the gradient is zero or does not exist.
- Convex function
- A convex function lies on or below every chord joining two points on its graph.
- Momentum
- Momentum accumulates past gradients in a velocity vector and steps along that velocity instead of the raw gradient.
Gradient Descent and Optimisation Geometry FAQ
How do I tell from a loss curve whether the learning rate is too big?
A rate that is too large makes the loss oscillate, rise or diverge, while a rate that is too small gives a slow, steady decline. A good rate falls quickly and then levels off. If two curves start from the same loss, they share an initialisation, so differences come from the step size.
Does a zero gradient mean I have found the minimum?
Not necessarily. Minima, maxima and saddle points all have zero gradient, so a stationary point needs further checking. For convex losses such as those of linear or logistic regression every local minimum is global, but networks with hidden units are not convex and gradient descent offers no guarantee there.
Why does gradient descent zigzag in a narrow valley?
The gradient is steep across the valley and gentle along it, so one learning rate overshoots in the steep direction while creeping in the shallow one. The path bounces from wall to wall. Momentum helps because opposing components cancel across steps while consistent ones accumulate.
What does momentum add to plain gradient descent?
Momentum keeps a running velocity that adds each new gradient to a fraction β of the previous velocity, then steps along the velocity. Components that point the same way step after step accumulate speed, while components that flip sign across a narrow valley partly cancel, which damps the zigzag that plain gradient descent shows.
Why not simply search a grid of parameter values?
The number of grid points grows exponentially with the number of parameters, so even a coarse grid is impossible for a network with millions of weights, and a grid can step straight over a narrow minimum. Gradient descent instead uses the local slope to choose a direction, at a cost that grows only linearly with the parameter count.
How should I choose when to stop gradient descent?
The notes list several stopping tests: the gradient norm falling below a tolerance, the change in the loss becoming tiny, the change in the parameters becoming tiny, or a fixed iteration budget running out. The function value can flatten before the parameters settle, so a gradient-norm test is often the more reliable signal.
Exam move
Practise the update rule on one-dimensional quadratics until you can predict the behaviour from the factor 1 − η f″ alone, then repeat on a two-variable bowl with different curvatures in each direction and sketch the path. Compute gradients at a point for mixed functions, such as products of polynomials with sine or tanh, writing each partial derivative separately.
Finally, draw loss-against-iteration curves for a small, a good and a large learning rate and annotate what each shape tells you. For the convergence-rate vocabulary, attach one concrete example to each term: an error of 1/k is sublinear, an error shrinking by 0.6 each step is linear, and a ratio that falls to zero is superlinear.
In written answers, name the rate and justify it from the error sequence rather than from a remembered label.
Working through Gradient Descent and Optimisation Geometry in ECE5179? Sia is AskSia’s AI Computer Science tutor — ask any ECE5179 Gradient Descent and Optimisation Geometry question and get a clear, step-by-step explanation grounded in how ECE5179 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
