MATH7005 Chap.6 Backpropagation Through Layers
Backpropagation Through Layers
Influence traced in reverse
Backpropagation reuses the forward computation graph to organise the chain rule. An upstream gradient arrives from the loss, each operation contributes a local derivative and shared paths add their contributions.
The chapter begins with a scalar graph before restoring matrix shapes and automatic differentiation.
What to verify
Store forward intermediates, label derivative directions and compare a selected analytic or automatic gradient with a centred finite difference.
Layerwise gradient norms then show where signals vanish, explode or disappear from the graph.
Paths multiply and branches add
A valid chain links neighbouring derivatives through a shared intermediate variable. When one value influences the loss through several later operations, calculate every path contribution and sum them at the branch point.
Residual connections create a direct identity route alongside a transformed route, and omitting either produces an incomplete gradient.
Batch reductions affect magnitude
Every record contributes to parameter gradients. A summed loss adds those contributions, while a mean loss divides by batch size. The two objectives have the same minimiser under fixed conditions but produce differently scaled updates.
Match the backward interpretation to the reduction used by the loss function before comparing learning rates.
Automatic differentiation still needs tests
A framework follows the graph it was given, including accidental detachments, in-place modifications and hard decisions. Check for missing and non-finite gradients, clear accumulation buffers at the intended time and finite-difference a small parameter.
Layerwise norms locate where a problem begins more effectively than one global gradient statistic.
What this chapter covers
- 01
Computation graph
- 02
Upstream gradient
- 03
Local derivative
- 04
Chain-rule product
- 05
Branch accumulation
- 06
Finite-difference check
Worked example · free
Differentiate a short scalar chain
- 2Evaluate the forward values.
- 3Write each local derivative.
- 2Multiply along the connected path.
Key terms
- Computation Graph
- A directed record of operations and intermediate values linking inputs to a final result.
- Upstream Gradient
- The sensitivity of the final loss to the current operation output.
- Local Derivative
- The derivative of one operation output with respect to its direct input.
- Gradient Check
- A comparison between a computed derivative and a small finite-difference approximation.
- Vanishing Gradient
- A gradient signal that becomes extremely small as it passes through earlier layers.
Backpropagation Through Layers FAQ
What does backpropagation add to the chain rule?
It organises and reuses intermediate derivatives across a computation graph, avoiding repeated calculation of the same subexpressions. The mathematics remains the chain rule, applied efficiently in reverse order.
Why store pre-activations during the forward pass?
Activation derivatives are evaluated at those pre-activation values. Discarding them forces recomputation or makes the backward result depend on an incorrect value, especially for gates such as ReLU.
Why perform a finite-difference check?
It offers an independent numerical approximation for a selected parameter. Close agreement supports the backward implementation, while disagreement exposes a sign, graph or stale-value error before a full training run.
What happens when a value feeds two later operations?
The loss depends on that value through both paths, so backpropagation computes each path contribution and adds them at the shared node. Overwriting one branch gives an incomplete gradient even when every local derivative is correct.
Why can gradients double between identical batches?
Many frameworks accumulate into existing gradient buffers. If those buffers are not cleared at the intended time, a repeated batch adds another contribution. Deliberate accumulation is valid, but its reduction and update schedule must be documented.
What does a layerwise gradient norm reveal?
It locates where the backward signal becomes tiny, explosive or absent. Compare successive layers and several batches rather than one global value, then connect the pattern to activation saturation, disconnected paths or unstable updates.
Why use a centred finite difference?
Evaluating the loss on both sides of a parameter cancels more first-order approximation error than a one-sided check. Choose a small but representable step and compare only a few coordinates because numerical checking is expensive.
Assessment move
Draw a scalar graph with one branch and one shared parameter. Place forward values above the nodes, derivative labels on edges and accumulated gradients below the nodes. Work backward from the loss without skipping variables, then perturb one parameter in both directions to obtain a centred finite-difference check. Extend the ledger to a dense layer by writing activation, weight and delta shapes.
In code, verify that gradient buffers are cleared, print missing or non-finite values and compare layerwise norms. Add a residual route and explain why its identity contribution is summed with the transformed contribution. Keep one tiny gradient-check case in the project tests so later refactoring cannot silently break the computation graph.
Working through Backpropagation Through Layers in MATH7005? Sia is AskSia’s AI Computer Science tutor — ask any MATH7005 Backpropagation Through Layers question and get a clear, step-by-step explanation grounded in how MATH7005 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
