The Hong Kong University of Science and Technology · FACULTY OF COMPUTER SCIENCE

MAIE5421 Chap.8 Training a Multilayer Perceptron

- one subject, every graph, every model, every mark
8 Chapters4-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 8 of 10 · MAIE5421

Training a Multilayer Perceptron

A journey through one simple model

The deep learning lecture teaches training through a single task, classifying MNIST handwritten digits, and a single model family, the multilayer perceptron. Each digit is a 28 by 28 greyscale image flattened to 784 pixel values between 0 and 255, and the network must output a probability for each of the ten digit classes.

Working through that one example, the lecture answers three questions in turn: what the model is, how its predictions are graded and how it improves.

The model: perceptrons, layers and activations

A perceptron weights its inputs, adds a bias and applies an activation function, and the weights are what training learns.

Layers of perceptrons, each fully connected to the next, form the multilayer perceptron, with hidden layers that progressively transform the input. Activations supply the non-linearity that makes depth useful: sigmoid for binary outputs, ReLU in hidden layers for its efficiency and simple gradient, and softmax at the output to produce class probabilities that sum to one.

The lecture's design uses three hidden layers of 512 ReLU units and a ten-way softmax output.

The grading: loss functions

A loss function measures how far predictions are from targets and guides how the weights change. Mean squared error serves regression.

Cross-entropy serves multi-class classification against a one-hot target, and its value is the negative logarithm of the probability given to the true class, so confident mistakes are punished heavily.

Binary cross-entropy with sigmoid outputs serves multi-label problems.

The improvement: backpropagation and optimisers

Training proceeds in batches; one batch update is an iteration and one pass over all batches is an epoch. Each iteration runs a forward pass, computes the loss, differentiates it with respect to every weight through backpropagation, and updates the weights with an optimiser.

Stochastic gradient descent subtracts the learning rate times the gradient, follows a noisy path, and is sensitive to the learning rate, which can cause divergence if too large and slow progress if too small; Adam, Adagrad and RMSprop adapt the step automatically. Dropout, which randomly removes nodes during training, guards against overfitting when the model is large relative to the data.

In this chapter

What this chapter covers

  • 01

    The digit classification task and its input and output

  • 02

    Perceptron weights, bias and activation

  • 03

    Building a multilayer perceptron from fully connected layers

  • 04

    Sigmoid, ReLU and softmax activations

  • 05

    Forward propagation and model structure design

  • 06

    Mean squared error, cross-entropy and binary cross-entropy

  • 07

    Batches, iterations and epochs

  • 08

    Backpropagation, learning rate, optimisers and dropout

Worked example · free

Count the parameters in a fully connected digit classifier

Q [6 marks]. AskSia-authored practice. A digit classifier maps 784 inputs through three hidden layers of 512 units to a 10-way softmax. Count its parameters layer by layer, then recount after widening the first hidden layer to 1,024 units. The marks shown are an AskSia study allocation and are not the University's marking scheme.
The lecture's digit classifier as a stack of fully connected layers.The lecture's digit classifier as a stack of fully connected layers. Each arrow is one weight matrix with its biases, and the parameter count beside it is the input width times the output width plus one bias per output.Input (28×28 = 784)Linear + ReLU (512)Linear + ReLU (512)Linear + ReLU (512)Linear + Softmax (10)401,920 parameters262,656 parameters262,656 parameters5,130 parametersTotal 932,362
The lecture's digit classifier as a stack of fully connected layers. Each arrow is one weight matrix with its biases, and the parameter count beside it is the input width times the output width plus one bias per output.
  • 1Each layer has inputs × outputs weights plus one bias per output.
  • 2784 × 512 + 512 = 401,920; 512 × 512 + 512 = 262,656 for each of the next two layers; 512 × 10 + 10 = 5,130.
  • 1Total: 401,920 + 262,656 + 262,656 + 5,130 = 932,362.
  • 2Widened: 784 × 1,024 + 1,024 = 803,840 and 1,024 × 512 + 512 = 524,800, so the total becomes 803,840 + 524,800 + 262,656 + 5,130 = 1,596,426.
The original network has 932,362 parameters, and 43% of them sit in the first layer because it connects every one of the 784 pixels to every hidden unit. Widening that layer to 1,024 units raises the total to 1,596,426, since both the first weight matrix and the one after it grow. This is the cost the lecture later attributes to fully connected layers on images: parameters scale with the product of layer widths, which is what motivates weight sharing in convolutional networks.
Sia tip — Activation functions add no parameters. Count only the linear layers, and never forget the bias, one per output unit.
Glossary

Key terms

Perceptron
A unit that weights its inputs, adds a bias and applies an activation function to produce one output.
Multilayer Perceptron
A network of perceptrons arranged in fully connected layers, with input, hidden and output layers.
Activation Function
A non-linear operation applied to a unit's weighted sum, which lets stacked layers model complex relations.
ReLU
The activation that outputs the larger of its input and zero, favoured in hidden layers for its efficiency and simple gradient.
Softmax
The output activation that turns a vector of scores into probabilities between 0 and 1 that sum to one.
Epoch
One complete pass of every training batch through the model.
Learning Rate
The step size by which weights move against their gradient at each update.
Dropout
A training technique that randomly removes nodes and their connections to reduce overfitting.
FAQ

Training a Multilayer Perceptron FAQ

What is the difference between an iteration and an epoch?

An iteration is one update of the model using one batch of training data. An epoch is one full pass through every batch in the training set. With 50,000 training images and a batch size of 100, one epoch contains 500 iterations, and training for 20 epochs performs 10,000 weight updates in total.

Why is cross-entropy used instead of mean squared error for digit classification?

Cross-entropy measures the surprise of a prediction through the negative logarithm of the probability given to the true class, so it penalises confident wrong answers very heavily and works naturally with a softmax output and a one-hot target. Mean squared error is the lecture's choice for regression, where the target is a continuous value rather than a class.

What happens if the learning rate is wrong?

If it is too large, each update overshoots the minimum and the loss can grow without limit, so training diverges. If it is too small, the loss falls so slowly that training takes far longer than necessary. The lecture lists the need to pre-define the learning rate as a weakness of plain stochastic gradient descent and points to adaptive optimisers such as Adam.

Study strategy

Assessment move

Take the four-layer digit network and write, for each layer, its input width, output width, parameter count and activation. Then compute one softmax and one cross-entropy value by hand from three scores. These two exercises cover most of the arithmetic an assignment can ask about a fully connected network, and they make the code in Chapter 10 easy to read.

Working through Training a Multilayer Perceptron in MAIE5421? Sia is AskSia’s AI Computer Science tutor — ask any MAIE5421 Training a Multilayer Perceptron question and get a clear, step-by-step explanation grounded in how MAIE5421 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 13 of your subjects at The Hong Kong University of Science and Technology, plus 1,000+ more Bibles across the AskSia library.
Sia - your MAIE5421 tutor, unlimited, worked the way the exam marks it
The full 4-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works