The Hong Kong University of Science and Technology · FACULTY OF COMPUTER SCIENCE

MAIE5421 Chap.10 Building a PyTorch Training Pipeline

- one subject, every graph, every model, every mark
8 Chapters4-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 10 of 10 · MAIE5421

Building a PyTorch Training Pipeline

Six components of a PyTorch pipeline

The lecture's PyTorch walkthrough uses MNIST digit recognition to build a complete pipeline in six parts: load the data, define the model, train it, evaluate it, tie the parts together and make predictions.

At each stage a design question has to be answered, from what the dataset and learning target are, through the model structure, to which loss function, optimiser and training loop to use.

PyTorch is installed in Colab with a single pip command, and a notebook bundle accompanies the lecture.

Loading data one sample at a time

Because real datasets can be far larger than memory, the lecture defines a custom Dataset class rather than loading everything at once.

Its init method records the file path, the number of samples and a list of indices; its len method reports the dataset size; and its getitem method reads one line of a CSV file, scales the pixel values into the range 0 to 1 and returns them with the label.

A DataLoader then wraps the dataset and delivers batches, with the batch size and whether to shuffle as its two key settings, and random_split divides the data into training and validation sets.

Defining the model

Two styles are taught. The Sequential API lists layers in order and suits a straightforward chain of linear, activation and dropout layers.

The Functional API defines a class with an init method that creates the building blocks and a forward method that wires them together, which allows branches and concatenation.

The lecture also introduces the shapes of linear, LSTM, two-dimensional convolution and max-pooling layers, and the reshape operation that flattens feature maps.

Loss, optimiser, loop and prediction

Mean squared error, cross-entropy and binary cross-entropy are available as ready-made losses, and a custom loss can be written as a class with a forward method.

Optimisers such as SGD, RMSprop, Adam, Adadelta and Adagrad update the model's parameters. The training loop runs the forward pass, computes the loss, zeroes the gradients, backpropagates and steps the optimiser for every batch, evaluates on validation data without tracking gradients, and saves the best model. Prediction reloads the saved weights, switches to evaluation mode and runs a forward pass without gradients.

In this chapter

What this chapter covers

  • 01

    The six components of a PyTorch pipeline

  • 02

    Custom Dataset classes with init, len and getitem

  • 03

    DataLoader batching, shuffling and random_split

  • 04

    Sequential and Functional model definitions

  • 05

    Input and output shapes of common layers

  • 06

    Ready-made and custom loss functions

  • 07

    Optimisers and the training loop

  • 08

    Validation, saving the best model and prediction

Worked example · free

Put the PyTorch training loop for one batch in the right order

Q [6 marks]. AskSia-authored practice. These five statements form the body of a training loop but have been shuffled: optimizer.step(); pred = model(x); loss.backward(); loss = loss_fn(pred, y); optimizer.zero_grad(). Put them in order, say what each does in the pipeline, and explain what goes wrong if zero_grad is omitted. The marks shown are an AskSia study allocation and are not the University's marking scheme.
The PyTorch training pipeline as one loop.The PyTorch training pipeline as one loop. Samples flow from the file through the dataset and loader into the model, the loss is computed, and its gradient drives the optimizer step that updates the model before the next batch arrives.CSV fileDatasetDataLoaderModelLossOptimizer stepone samplebatchesbackward()update weights
The PyTorch training pipeline as one loop. Samples flow from the file through the dataset and loader into the model, the loss is computed, and its gradient drives the optimizer step that updates the model before the next batch arrives.
  • 1Forward pass first: pred = model(x).
  • 1Then the loss: loss = loss_fn(pred, y).
  • 2Clear old gradients with optimizer.zero_grad(), then compute new ones with loss.backward().
  • 2Finally update the weights with optimizer.step(); without zero_grad, gradients from earlier batches accumulate into every update.
The order is pred = model(x), loss = loss_fn(pred, y), optimizer.zero_grad(), loss.backward(), optimizer.step(). The first line runs the model on the batch, the second scores its prediction, the third clears the gradients left by the previous batch, the fourth backpropagates to fill each parameter's gradient, and the fifth moves every parameter against its gradient. Zero_grad may come anywhere before backward, but it must come before it: PyTorch adds new gradients to stored ones, so omitting it makes each update use the running sum of all previous gradients and training becomes erratic.
Sia tip — Read a training loop as forward, score, clear, backward, step. If any line is missing or out of order, the loss curve will tell you within a few iterations.
Glossary

Key terms

PyTorch
A free, open-source Python library for defining, training and evaluating deep learning models.
Dataset Class
A class with init, len and getitem methods that tells PyTorch how many samples exist and how to produce each one.
DataLoader
The PyTorch object that wraps a dataset and delivers batches of samples, optionally in shuffled order.
Batch Size
The number of samples processed together in one iteration of training.
Sequential API
A way of defining a network as an ordered list of layers applied one after another.
Functional API
A way of defining a network as a class whose forward method wires its building blocks together, allowing branches.
State Dict
The dictionary of a model's learned parameters, which is saved during training and loaded for prediction.
FAQ

Building a PyTorch Training Pipeline FAQ

What do the three methods of a PyTorch Dataset do?

The init method sets up whatever the other methods need, such as the file path, the number of samples and a list of indices. The len method returns the number of samples, which tells the loader when an epoch ends. The getitem method takes an index and returns one sample, typically the input features and the label, and it is where any per-sample conversion or normalisation happens.

Why does my training batch change every time I run the code?

Because the training loader has shuffle switched on, so it draws the samples in a new random order each epoch, and random_split also chooses the training and validation samples at random. That behaviour is intended, since shuffling helps training. If you need identical runs, for example to compare two models fairly in a report, fix the random seed before creating the split and the loaders.

Should my model end with a softmax layer if I use CrossEntropyLoss?

No. The lecture notes that CrossEntropyLoss passes its input through a softmax internally and expects the indices of the correct classes as targets. If the model already applies softmax, the scores are squashed twice and training slows. Return the raw outputs of the final linear layer for training, and apply softmax separately only when you need probabilities for display or thresholding.

Study strategy

Assessment move

Type the lecture's pipeline from memory in the order dataset, loader, model, loss, optimiser, loop, prediction, and run it on a small subset of the digit data. Then break it on purpose: remove zero_grad, add a softmax before CrossEntropyLoss, swap the shuffle settings. Watching what each mistake does to the loss is the fastest way to recognise it in your own project code.

Working through Building a PyTorch Training Pipeline in MAIE5421? Sia is AskSia’s AI Computer Science tutor — ask any MAIE5421 Building a PyTorch Training Pipeline question and get a clear, step-by-step explanation grounded in how MAIE5421 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 13 of your subjects at The Hong Kong University of Science and Technology, plus 1,000+ more Bibles across the AskSia library.
Sia - your MAIE5421 tutor, unlimited, worked the way the exam marks it
The full 4-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works