ECE5179 Chap.7 Residual Networks and Batch Normalisation
Residual Networks and Batch Normalisation
Week 7 revisits convolutional design with two ideas that made very deep networks trainable. Residual connections let a block output F(x) + x, so extra depth can default to the identity and gradients flow through the shortcut. Batch normalisation standardises features with mini-batch statistics and then rescales them with learned parameters, behaving differently in training and evaluation.
The chapter covers ResNet stages, bottleneck blocks, normalisation alternatives and the numbers behind them.
What this chapter covers
- 01
The degradation problem in deep plain networks
- 02
Residual blocks and identity shortcuts
- 03
Downsampling shortcuts and bottleneck blocks
- 04
ResNet-18 to ResNet-152 stage designs
- 05
Batch normalisation for fully connected and convolutional layers
- 06
Training mode versus evaluation mode
- 07
Group and layer normalisation
- 08
Why batch norm smooths the loss landscape
Worked example · free
Batch normalise a four-sample batch
- 1Batch mean μ = 20/4 = 5.
- 1Squared deviations 9, 1, 1, 9 give variance σ² = 5 and standard deviation about 2.236.
- 1Normalised values (x − 5)/2.236: −1.342, −0.447, 0.447, 1.342.
- 1Outputs 2x̂ + 1: −1.683, 0.106, 1.894, 3.683.
Key terms
- Degradation problem
- The degradation problem is the rise in training error seen when plain networks are made deeper.
- Bottleneck block
- A bottleneck block reduces channels with a 1 × 1 convolution, applies a 3 × 3 convolution, and restores the channels with another 1 × 1 convolution.
- Running statistics
- Running statistics are the stored averages of batch means and variances that batch normalisation uses at evaluation time.
- Group normalisation
- Group normalisation computes statistics over groups of channels within each sample, independent of the batch size.
Residual Networks and Batch Normalisation FAQ
Why do residual connections help very deep networks?
A residual block outputs F(x) + x, so learning the identity only requires F to be zero, which is easy. The derivative of the block includes an identity term, so gradients pass through the shortcut undiminished. The lectures show that plain deep networks underfit rather than overfit, so this is an optimisation fix.
Why does batch norm behave differently at test time?
During training it uses the statistics of the current mini-batch, so each output depends on the other samples. At evaluation it switches to stored running averages, so a single input always gets the same output. Forgetting model.eval() leaves the training behaviour on and makes test results noisy.
Why can a convolution placed before batch norm drop its bias?
Batch normalisation subtracts the mean of each feature, which removes any constant the convolution added, and its own β parameter supplies a learned shift instead. Keeping both biases wastes parameters without changing what the network can represent.
How does a ResNet change resolution between stages?
The first block of each stage halves the height and width with a stride-2 convolution and doubles the number of channels. Because the input no longer matches that shape, its shortcut becomes a 1 × 1 convolution with stride 2. Across ResNet-18 the maps shrink from 56 to 28, 14 and 7 pixels square.
What is the difference between batch, layer and group normalisation?
All three standardise activations and then apply a learned scale and shift; they differ in which values share statistics. Batch normalisation pools one feature across the batch, layer normalisation pools all features within one sample, and group normalisation pools groups of channels within one sample, so the last two do not depend on batch size.
Exam move
Draw a basic and a bottleneck residual block from memory, label the tensor shapes, and redraw the downsampling version with its 1 × 1 stride-2 shortcut. Count the weights in each for a chosen channel number to see why bottlenecks are used from ResNet-50 upwards.
Practise batch normalisation on small batches, including a shifted batch to confirm the output does not change, and write a two-sentence explanation of training versus evaluation mode, including the frozen-weights trap from the transfer-learning workshop.
For the theory question that often accompanies batch normalisation, prepare a short answer contrasting the original covariate-shift explanation with the smoother-loss-landscape view the lectures favour, and say what each predicts about learning rates.
Working through Residual Networks and Batch Normalisation in ECE5179? Sia is AskSia’s AI Computer Science tutor — ask any ECE5179 Residual Networks and Batch Normalisation question and get a clear, step-by-step explanation grounded in how ECE5179 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
