The Hong Kong University of Science and Technology · FACULTY OF COMPUTER SCIENCE

MAIE5421 Chap.9 Deep Architectures for Images and Sequences

- one subject, every graph, every model, every mark
8 Chapters5-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 9 of 10 · MAIE5421

Deep Architectures for Images and Sequences

Why images need more than fully connected layers

The lecture motivates convolutional networks with two weaknesses of the multilayer perceptron on images.

It treats an image as a flat vector, ignoring the two-dimensional layout of the pixels, and its fully connected layers multiply the number of parameters as images grow larger and networks deeper, without sharing weights between operations that do the same job in different places.

Convolution and pooling

A convolution layer computes each output value as the sum of element-wise products between a small kernel and a patch of the input, then applies an activation; the same kernel is reused at every position, so its weights are shared.

The kernel size sets how much input feeds each output, the stride sets how far the kernel moves and so how much the output shrinks, and the number of kernels sets the number of output channels. Stacked convolutions capture progressively higher-level structure.

Max and average pooling then shrink the feature maps, reducing redundancy and enlarging the receptive field, before the result is flattened for a final classifier.

Depth and residual connections

The ImageNet benchmark shows the payoff of depth: top-5 error fell from 28.2% with hand-designed features and support vector machines in 2010 to 4.5% with ResNet in 2015. Deeper networks are harder to train because gradients, multiplied across layers on the way back, fade before reaching the early layers.

ResNet's residual block adds a skip connection around a pair of layers, so the layers learn only a residual relative to their input and the gradient gains a direct path backwards.

Variants from ResNet-18 to ResNet-152 trade capacity against computation.

Sequences, attention and transfer

Recurrent networks share weights across time steps and carry a state from one step to the next, and the LSTM uses forget, input and output gates around a cell state to control what is kept. Their step-by-step processing is slow and struggles with long contexts.

The Transformer uses attention to gather the relevant parts of a sequence directly, as in translation, where each output word can focus on the input words that matter most. Transfer learning closes the lecture: a model pretrained on large generic data, such as VGG, ResNet or BERT, is fine-tuned on a small task-specific dataset.

In this chapter

What this chapter covers

  • 01

    Why flat, fully connected networks suit images poorly

  • 02

    Convolution kernels, kernel size, stride and channels

  • 03

    Max and average pooling and the receptive field

  • 04

    The ImageNet error record from 2010 to 2015

  • 05

    Gradient vanishing in very deep networks

  • 06

    Residual blocks and skip connections

  • 07

    Recurrent networks and LSTM gates

  • 08

    Attention, the Transformer and transfer learning

Worked example · free

Trace one input through a residual block and see why its gradient survives

Q [6 marks]. AskSia-authored practice. A residual block receives x = (1.0, -2.0), and its two weight layers produce F(x) = (0.3, 0.5). Compute the block's output, then compare how much of a gradient reaches the first of 20 stacked layers when every layer scales it by 0.5, with and without an identity skip path. The marks shown are an AskSia study allocation and are not the University's marking scheme.
A residual block.A residual block. The input travels through two weight layers to give F(x) and, unchanged, along the orange skip path; the two are added before the final ReLU, so the layers only have to learn the difference from the input.Weight layerReLUWeight layerInput x+Skip: identity xF(x)ReLU(F(x) + x)
A residual block. The input travels through two weight layers to give F(x) and, unchanged, along the orange skip path; the two are added before the final ReLU, so the layers only have to learn the difference from the input.
  • 2Add the skip path: F(x) + x = (0.3 + 1.0, 0.5 - 2.0) = (1.3, -1.5).
  • 1Apply ReLU: (1.3, 0), since the negative entry is set to zero.
  • 2Plain stack: 0.5 multiplied 20 times is about 0.00000095, so the early layer barely learns.
  • 1With a skip path the identity contributes a derivative of 1 at every block, so the gradient no longer has to pass through the shrinking product alone.
The block outputs (1.3, 0): the skip path adds the input back to the residual, and the final ReLU clips the negative component. In a plain 20-layer stack that scales the gradient by 0.5 at each layer, the first layer receives roughly one millionth of the gradient at the output, which is what the lecture calls gradient vanishing. The identity path in each residual block passes the gradient back unchanged alongside the layers, which is the highway that lets networks such as ResNet-152 train at all.
Sia tip — When asked why ResNet works, name the mechanism: the skip connection makes the block learn a difference from its input and gives the gradient a direct path back.
Glossary

Key terms

Convolution Kernel
A small grid of learned weights slid across an input, whose element-wise products with each patch are summed into one output value.
Stride
The number of pixels a kernel moves between applications; a stride above one shrinks the output.
Receptive Field
The region of the original input that influences one value in a later feature map.
Max Pooling
Downsampling that keeps the largest value in each small window of a feature map.
Gradient Vanishing
The fading of gradients as they are multiplied back through many layers, which stops early layers from learning.
Skip Connection
A path that carries a block's input around its layers and adds it to their output.
LSTM
A recurrent network that uses forget, input and output gates around a cell state to control information over a sequence.
Fine-tuning
Continuing to train a pretrained model on a small dataset for the specific task at hand.
FAQ

Deep Architectures for Images and Sequences FAQ

Why do convolutional networks need fewer parameters than fully connected ones?

Because a convolution kernel is reused at every position of the image. A 3 by 3 kernel has nine weights per input channel however large the image is, while a fully connected layer needs a separate weight for every pair of input and output units. Pooling then shrinks the feature maps, so the final fully connected layer sees far fewer values than the raw pixels.

What problem does ResNet solve?

Gradient vanishing in very deep networks. As gradients are multiplied back through many layers they shrink until the early layers stop updating. ResNet's residual blocks add the input of each block to its output through a skip connection, so the layers learn only a residual and the gradient has a direct path backwards. That is what let ResNet reach a top-5 ImageNet error of 4.5% in 2015.

When should I use transfer learning in a project?

Whenever your labelled dataset is small compared with what a deep model needs. Start from a network pretrained on a large generic dataset, such as a ResNet or VGG trained on a large image collection, and fine-tune it on your data. The pretrained weights already encode useful visual features, so you need fewer labelled examples and less training time, and overfitting is less likely.

Study strategy

Assessment move

For every architecture in the chapter, write one sentence naming the problem it solves and the mechanism it uses: convolution for weight sharing over space, pooling for shrinking maps, residual blocks for gradient flow, recurrence for weight sharing over time, attention for direct access to relevant inputs, and fine-tuning for small datasets. Then trace tensor shapes through one small CNN and one LSTM by hand.

Working through Deep Architectures for Images and Sequences in MAIE5421? Sia is AskSia’s AI Computer Science tutor — ask any MAIE5421 Deep Architectures for Images and Sequences question and get a clear, step-by-step explanation grounded in how MAIE5421 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 13 of your subjects at The Hong Kong University of Science and Technology, plus 1,000+ more Bibles across the AskSia library.
Sia - your MAIE5421 tutor, unlimited, worked the way the exam marks it
The full 5-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works