ELEC5622 Chap.12 Neural Networks: MLP Training and CNNs
Neural Networks: MLP Training and CNNs
This chapter covers the neural network foundations taught in Week 10. It explains why a single layer cannot solve XOR, why hidden layers must be non-linear, and how least-squares and cross-entropy losses are minimised by gradient descent. Backpropagation applies the chain rule layer by layer, and convolutional networks use local receptive fields and shared weights for images.
A worked example pushes one sample forward and backward through a ReLU unit. These foundations lead into the deep-learning classification and segmentation lectures and Project 2 later in the semester. The lecture's tiny ReLU network is worked forward and backward with numbers, so every gradient can be checked.
What this chapter covers
- 01
From hand-crafted features to end-to-end learning
- 02
Linear separability and the XOR problem
- 03
Activation functions and their derivatives
- 04
Least-squares and cross-entropy losses
- 05
Forward and backward phases of training
- 06
Learning rate, batches and momentum
- 07
Weight initialisation
- 08
Receptive fields and weight sharing in CNNs
Worked example · free
One gradient descent step on a quadratic loss
- 1J′(w) = 2(w − 2), so J′(0) = −4.
- 1w1 = 0 − 0.1 × (−4) = 0.4.
- 1J falls from (0 − 2)² = 4 to (0.4 − 2)² = 2.56.
Key terms
- Backpropagation
- Backpropagation computes every weight gradient by applying the chain rule backwards from the loss through each layer.
- Learning rate
- The learning rate is the step size that scales each gradient descent update of the weights.
- Receptive field
- A receptive field is the local region of the input grid that one convolutional neuron is connected to.
- Mini-batch
- A mini-batch is the small fixed set of training samples used to estimate the gradient in one update.
- Activation function
- An activation function is the non-linear map applied to a neuron's weighted input sum.
- Weight sharing
- Weight sharing reuses one set of filter weights at every position of a convolutional layer.
Neural Networks: MLP Training and CNNs FAQ
Why can't a single-layer network learn XOR?
XOR's positive and negative cases cannot be separated by one straight line: the inequalities a single linear unit would need contradict each other. A hidden layer with non-linear activations lets the network carve a boundary that no single line can draw.
What goes wrong if the learning rate is too large?
Updates overshoot the minimum, and if the rate is large enough each step lands further away, so the loss grows and training diverges. A rate that is too small is safe but needs many more updates to converge.
How is an iteration different from an epoch?
An iteration is one forward and backward pass on one mini-batch. An epoch is one full pass of the whole training set, so 1,000 samples in batches of 50 need 20 iterations per epoch.
Why do stacked linear layers not add power?
Composing linear maps gives another linear map, so a network with only linear hidden units computes the same family of functions as a single layer. Non-linear activations such as ReLU are what let extra layers represent new boundaries.
Why does weight sharing suit images?
A useful pattern such as an edge can appear anywhere in an image. Sharing one filter's weights across all positions detects it everywhere with few parameters and makes the response independent of location.
Why does weight initialisation matter?
The lecture warns that weights that start too small make activations shrink towards zero, while weights that start too large make them explode. Schemes such as Xavier initialisation, scaled by the number of input neurons, and Kaiming initialisation keep signals in a workable range.
Exam move
Work the lecture's small ReLU network by hand with your own numbers until you can produce every gradient in order. Learn the three activation derivatives and when ReLU blocks the gradient. Practise iteration and epoch counts, then explain sparse connectivity and weight sharing in CNNs in your own words. Close by comparing batch, stochastic and mini-batch gradient descent on cost and gradient noise, in one line each.
Write the gradient descent update and the chain rule on a blank page from memory, then use them on a fresh two-weight network of your own design.
Working through Neural Networks: MLP Training and CNNs in ELEC5622? Sia is AskSia’s AI Electrical Engineering tutor — ask any ELEC5622 Neural Networks: MLP Training and CNNs question and get a clear, step-by-step explanation grounded in how ELEC5622 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
