ECE5179 Chap.5 Convolutional Neural Networks
Convolutional Neural Networks
Weeks 5 and 6 introduce convolutional networks for images. A small kernel slides over the input with shared weights, so the layer detects local patterns anywhere while using far fewer parameters than a dense layer.
The chapter gives the output-size, parameter and multiplication formulas used throughout the studios, works a convolution by hand, covers padding, stride, dilation and pooling, and traces the design line from LeNet-5 to AlexNet and into PyTorch.
What this chapter covers
- 01
Why convolution suits images: locality and shared weights
- 02
Output size with padding, stride and dilation
- 03
Parameter and multiplication counts
- 04
Pooling and receptive fields
- 05
LeNet-5 and AlexNet
- 06
Transposed and depthwise-separable convolutions
- 07
Convolution as a matrix and in PyTorch
- 08
AlexNet's memory, parameter and multiplication profile
Worked example · free
Slide a 3 × 3 kernel over a 5 × 5 input
- 1Output size: (5 − 3)/1 + 1 = 3, so the result is 3 × 3.
- 1The kernel adds the left column of each window and subtracts the right column.
- 1Top-left window: left column sums to 3 and right column to 5, so the first entry is −2.
- 1Sliding across and down gives rows (−2, 0, 2), (−1, −1, 0) and (1, 0, −1).
- 1Each entry uses 9 multiplications and there are 9 entries, so 81 multiplications in total.
Key terms
- Kernel
- A kernel is the small grid of shared weights that a convolutional layer slides across its input.
- Stride
- Stride is the number of positions the kernel moves between consecutive outputs.
- Zero padding
- Zero padding adds rows and columns of zeros around an input so the kernel can be centred on edge pixels.
- Pooling
- Pooling summarises each small neighbourhood of a feature map by its maximum or average, with no learned weights.
- Dilation
- Dilation spaces out the taps of a kernel so it covers a wider area with the same number of weights.
Convolutional Neural Networks FAQ
What formula gives the output size of a convolution?
With input size H, padding p on each side, kernel size k, dilation d and stride s, the output size is the floor of (H + 2p − d(k − 1) − 1)/s, plus one. With no dilation this becomes the floor of (H + 2p − k)/s plus one, which covers most studio questions.
How are multiplications counted for a convolutional layer?
The unit counts one multiplication per kernel weight per output value, which equals the multiply-accumulate count. A layer therefore needs C_out × H_out × W_out × C_in × k × k multiplications. Pooling layers have no weights, and parameter counts never depend on the image size.
Why did AlexNet matter?
AlexNet won the 2012 ImageNet challenge by scaling up the LeNet design. It replaced sigmoid with ReLU, average pooling with max pooling and added dropout in its fully connected layers, which hold almost all of its parameters, while most of its computation sits in the convolutional layers.
What does padding do to the output size?
Padding adds p rows and columns of zeros on each side before the kernel slides, increasing the effective input to H + 2p. With stride 1 and an odd kernel size k, choosing p = (k − 1)/2 keeps the output the same size as the input, which is why 3 × 3 kernels are usually paired with padding 1.
Why is a convolutional layer cheaper than a dense layer on images?
A dense layer connects every pixel to every output unit, so its weight count grows with the image size. A convolutional layer reuses one small kernel at every position, so its parameters depend only on channels and kernel size. The Studio 8 classifier shows the effect with about nine times fewer parameters than an MLP.
Exam move
Drill the three formulas, output size, parameters and multiplications, on new layer settings until each takes less than a minute, always writing the output size first.
Work at least one convolution and one pooling operation completely by hand, including a version with padding and one with stride 2. Then fill a layer-by-layer table for a small network, as the AlexNet slides do, with columns for output shape, parameters and multiplications, and check that the parameter column does not change if the image size does.
Finally, connect the formulas to design choices: say what stride 2 buys in computation, what dilation buys in receptive field, and what a 1 × 1 convolution does to channels, because short explanation questions often follow the arithmetic.
Working through Convolutional Neural Networks in ECE5179? Sia is AskSia’s AI Computer Science tutor — ask any ECE5179 Convolutional Neural Networks question and get a clear, step-by-step explanation grounded in how ECE5179 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
