ECE5179 Neural networks and deep learning

How ECE5179 is assessed

Weights are from the unit's Assessment Summary and agree with the Monash Handbook entry for ECE5179 in 2026 (quizzes 5%, test 15%, written work 30%, examination 50%). The unit materials state a pass requirement of an overall mark of at least 50%. The final examination's duration, question format and permitted materials are not given in the unit pages available here; check the examination notice on Moodle and your official timetable.

Current ECE5179 dates

Dates are as published in the ECE5179 Moodle schedule and assessment pages for Semester 2 2026. Confirm exact deadlines and submission settings in the live LMS.

What ECE5179 covers

Follow the unit from linear models and gradient descent through multilayer perceptrons, convolutional, residual, generative and recurrent networks to attention, language models and vision transformers.

The unit synopsis describes neural networks first, then how they are trained with backpropagation, then the main families of deep networks: multilayer perceptrons, convolutional networks and recurrent networks, followed by modern transformers and large language models, and how such models are deployed and fine-tuned.

Optimisation and generalisation theory explains why these networks behave as they do, and Python with PyTorch is the programming environment.

The teaching runs over twelve weeks of workshops and studios. Weeks 1 to 3 cover regression and classification models and gradient descent.

Week 4 introduces multilayer perceptrons and the backpropagation algorithm, Week 5 convolutional networks, and Week 6 PyTorch alongside the mid-semester test.

Week 7 revisits convolutional design with residual networks and batch normalisation, Week 8 turns to generative adversarial networks, and Week 9 to recurrent networks and LSTMs. The final three weeks cover the attention mechanism, large language models and transformers for text and images.

Assessment combines continuous work worth half the unit with a final examination worth the other half.

The in-semester tasks are ten engagement quizzes, two homeworks that mix coding with written questions, an in-person mid-semester test and an individual assignment. To pass, a student needs an overall mark of at least 50%.

Studio worksheets carry no marks but come with released solutions, and they are the best rehearsal for the hand calculations the tests reward: gradients, layer sizes, attention weights and losses worked step by step.

Worked example · free

Gradient descent zigzags on an elongated bowl

Key terms

Mean squared error

Mean squared error is the average of the squared differences between predictions and targets, the loss minimised by least-squares regression.

Sigmoid function

The sigmoid function σ(z) = 1/(1 + e^(−z)) maps any real score to a probability between 0 and 1.

Cross-entropy loss

Cross-entropy loss is the negative log-probability a model assigns to the true class, averaged over the training samples.

Softmax

Softmax converts a vector of logits into probabilities by exponentiating each entry and dividing by the sum of the exponentials.

Learning rate

The learning rate is the step size that scales the negative gradient in each gradient-descent update.

Backpropagation

Backpropagation is the algorithm that applies the chain rule backwards through a computation graph to find how the loss changes with each parameter.

Receptive field

A receptive field is the region of the input image that influences one value in a convolutional feature map.

Dropout

Dropout is a regulariser that sets each hidden unit to zero with probability p during training and rescales the survivors.

Residual connection

A residual connection adds a block's input to its output, so the block computes F(x) + x and can represent the identity easily.

Batch normalisation

Batch normalisation standardises each feature using mini-batch statistics and then applies a learned scale and shift.

Mode collapse

Mode collapse is a GAN failure in which the generator produces realistic samples that cover only part of the real data distribution.

LSTM cell

An LSTM cell is a gated recurrent unit with a separate cell state controlled by forget, write and output gates.

Scaled dot-product attention

Scaled dot-product attention weights value vectors by a softmax of query-key dot products divided by the square root of the key dimension.

ECE5179 FAQ

Questions students ask in each ECE5179 chapter

Why does the least-squares line pass through the mean point?

Setting the derivative of the mean squared error with respect to the intercept to zero forces the residuals to average zero. That condition gives b = ȳ − w x̄, and substituting x̄ into the line returns ȳ exactly. Checking this property is a quick way to catch an arithmetic slip in a hand-fitted line. Chapter 01 · Learning Types and Linear Regression

Why use cross-entropy instead of squared error for classification?

With a sigmoid output, squared error keeps a factor ŷ(1 − ŷ) in its gradient, so a confidently wrong prediction learns about a thousand times more slowly than under cross-entropy. Cross-entropy is also convex in the weights, so gradient descent has no poor local minima to fall into for this model. Chapter 02 · Logistic Regression and the Softmax Classifier

How do I tell from a loss curve whether the learning rate is too big?

A rate that is too large makes the loss oscillate, rise or diverge, while a rate that is too small gives a slow, steady decline. A good rate falls quickly and then levels off. If two curves start from the same loss, they share an initialisation, so differences come from the step size. Chapter 03 · Gradient Descent and Optimisation Geometry

Why do neural networks need activation functions?

Without them, each layer is an affine map and a composition of affine maps is still affine, so the whole network can only draw flat decision boundaries. The non-linearity between layers is what lets hidden layers bend the input space and build features that a final linear layer can separate. Chapter 04 · Multilayer Perceptrons and Backpropagation

What formula gives the output size of a convolution?

With input size H, padding p on each side, kernel size k, dilation d and stride s, the output size is the floor of (H + 2p − d(k − 1) − 1)/s, plus one. With no dilation this becomes the floor of (H + 2p − k)/s plus one, which covers most studio questions. Chapter 05 · Convolutional Neural Networks

Why must survivors be rescaled in dropout?

If units are zeroed with probability p during training but all fire at test time, the next layer receives more input at test time than it learned to expect. Inverted dropout divides the surviving activations by 1 − p during training, so the expected input is the same in both modes and the test network needs no change. Chapter 06 · Overfitting and Regularisation

Why do residual connections help very deep networks?

A residual block outputs F(x) + x, so learning the identity only requires F to be zero, which is easy. The derivative of the block includes an identity term, so gradients pass through the shortcut undiminished. The lectures show that plain deep networks underfit rather than overfit, so this is an optimisation fix. Chapter 07 · Residual Networks and Batch Normalisation

Why is the non-saturating generator loss preferred?

Early in training the discriminator rejects fakes confidently, so D(G(z)) is near zero and the minimax loss log(1 − D(G(z))) is almost flat. Measured against the discriminator's logit, its gradient is −D, which is tiny, while the non-saturating loss −log D(G(z)) gives −(1 − D), close to −1 in the same direction. Chapter 08 · Generative Adversarial Networks

Why do vanilla RNNs forget long-range information?

Backpropagating across k steps multiplies k factors, each the recurrent weight times a tanh derivative of at most one. When those factors are below one the product shrinks geometrically, so early inputs barely affect the gradient. Factors above one make it explode instead, which clipping can control but vanishing cannot be fixed that way. Chapter 09 · Recurrent Neural Networks and LSTMs

Why divide the attention scores by the square root of d_k?

If query and key entries are roughly independent with unit variance, their dot product has variance close to d_k. Large scores push softmax towards a one-hot output whose gradients nearly vanish. Dividing by √d_k restores unit scale, which the lectures describe as scaling the softmax for stability. Chapter 10 · Attention and the Transformer Block

Why do transformers need positional encodings?

Dot-product attention is permutation equivariant: shuffling the input tokens simply shuffles the outputs in the same way. Two sentences with the same words in a different order would therefore look identical. Adding a position vector to each token makes order part of the input, so attention can use it. Chapter 11 · Tokenisation, Positional Encoding and BERT

How many tokens does a Vision Transformer read?

For an H by W image and patch size P there are (H/P) times (W/P) patch tokens, plus one classification token. A 224 by 224 image with 16 by 16 patches gives 196 patches and 197 tokens. Halving the patch size quadruples the patches and multiplies the attention table roughly sixteen-fold. Chapter 12 · Vision Transformers, Evaluation and Decoding

How to study for the exam

Work through the unit in its own order, because every later week reuses an earlier one: least squares and cross-entropy become output layers, gradient descent becomes the training loop, and backpropagation is the same chain rule on a larger graph. For each week, rebuild the key derivation on a blank page before reading the notes again, then redo one studio problem without the solution and compare line by line.

Keep a single sheet of size rules, covering parameter counts for dense, convolutional and recurrent layers, convolution output size, multiplication counts and attention shapes, and test yourself on new numbers until each takes under a minute. When a calculation goes wrong, record whether the cause was a sign, a shape, a missed factor or a calculator mode, since the same slip tends to recur.

In the final fortnight, mix questions from different weeks so the topic name no longer tells you the method, and practise explaining a design choice in two or three sentences after every calculation.

Your AI Computer Science tutor for ECE5179

Stuck on a hard ECE5179 question? Sia is AskSia’s AI Computer Science tutor — ask any ECE5179 Neural networks and deep learning question and get a clear, step-by-step explanation grounded in how the course is actually taught and assessed. Read this whole study guide free, then take your hardest questions to Sia.