ECE5179 Chap.4 Multilayer Perceptrons and Backpropagation
Multilayer Perceptrons and Backpropagation
Week 4 builds the first deep networks. A multilayer perceptron alternates affine maps with non-linear activations; without the activations, any stack of layers collapses to a single linear map.
The chapter counts parameters, states what such networks can represent, and then derives backpropagation as the chain rule applied backwards through a computation graph, with a full forward and backward pass and the constructive design questions that recur in tests.
What this chapter covers
- 01
Why stacked linear layers collapse
- 02
Sigmoid, tanh, ReLU and smoother activations
- 03
MLP anatomy and parameter counts
- 04
Universal approximation and the role of depth
- 05
The chain rule on a computation graph
- 06
Local gradient rules for linear, ReLU and sigmoid nodes
- 07
Building Boolean functions and regions from simple units
- 08
Training loops in PyTorch and the Adam optimiser
Worked example · free
Forward values and local gradients on a computation graph
- 1Forward values: a = 0, h = σ(0) = 0.5, z = 1.5 and L = 0.125.
- 1Seed the backward pass: ∂L/∂z = z − 1 = 0.5.
- 1Output weight: ∂L/∂v = 0.5 × h = 0.25.
- 1Into the hidden unit: ∂L/∂h = 0.5 × v = 1.5, then ∂L/∂a = 1.5 × σ(0)(1 − σ(0)) = 0.375.
- 1First weight: ∂L/∂w1 = 0.375 × x1 = 0.375.
- 1Second weight: ∂L/∂w2 = 0.375 × x2 = −0.75.
Key terms
- Activation function
- An activation function is the fixed non-linearity applied elementwise after each affine layer of a network.
- ReLU
- ReLU outputs its input when positive and zero otherwise, so its derivative is 1 or 0.
- Universal approximation theorem
- The universal approximation theorem says a sufficiently wide single hidden layer can come arbitrarily close to every continuous function on a compact set.
- Computation graph
- A computation graph records each operation of a model as a node so that derivatives can be propagated along its edges.
- Vanishing gradient
- A vanishing gradient is a gradient that shrinks towards zero as it is multiplied through many layers.
- Dying ReLU
- A dying ReLU is a unit whose input stays negative, so it outputs zero and passes no gradient.
Multilayer Perceptrons and Backpropagation FAQ
Why do neural networks need activation functions?
Without them, each layer is an affine map and a composition of affine maps is still affine, so the whole network can only draw flat decision boundaries. The non-linearity between layers is what lets hidden layers bend the input space and build features that a final linear layer can separate.
How do I count the parameters of an MLP?
Each layer from n inputs to m outputs has n times m weights plus m biases. Add the layers together. For example, 12 to 16 to 16 to 4 holds 208 plus 272 plus 68, which is 548 parameters. Activation functions add none.
What does backpropagation actually compute?
It finds how the loss changes with each weight in one backward sweep, multiplying local derivatives along each path from the loss and adding contributions where paths meet. Its cost is roughly one extra forward pass, which is why gradient descent on large networks is practical.
Why can a single ReLU unit not compute XOR?
A single unit thresholds one linear score, so its positive region is one side of a line. XOR is positive on two opposite corners of the square and negative on the other two, which no single line can separate. One hidden layer of two or more units fixes this by combining two boundaries.
What does the universal approximation theorem not tell you?
It guarantees that a sufficiently wide single hidden layer can come arbitrarily close to every continuous function on a bounded region, but it does not say how wide that layer must be, whether gradient descent will find the right weights, or whether the result will generalise to new data. Depth often achieves the same accuracy far more economically.
Exam move
Draw the computation graph before writing any derivative, then fill a forward table and a backward table side by side. Practise on graphs with a branch, where a value feeds two later nodes, so the add-where-arrows-meet rule becomes automatic. Count parameters for several made-up architectures and confirm each count by a second method.
For design questions, rehearse the ReLU identities for absolute value and identity and the pattern-detector construction for Boolean functions. When a question asks for a gradient with respect to an input rather than a weight, use the same graph and swap the final factor: ∂L/∂x comes from Wᵀ times the upstream gradient.
Practising both directions on the same small network is the quickest way to make the linear-layer rules automatic.
Working through Multilayer Perceptrons and Backpropagation in ECE5179? Sia is AskSia’s AI Computer Science tutor — ask any ECE5179 Multilayer Perceptrons and Backpropagation question and get a clear, step-by-step explanation grounded in how ECE5179 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
