COMP4318 · Machine Learning and Data Mining
Deep Neural Networks: CNNs, RNNs & Transformers
Weeks 8–9 cover deep learning: convolutional neural networks (CNNs) for images, recurrent networks (RNNs/LSTM/GRU) for sequences, and the transformer/attention architecture, along with softmax and cross-entropy. Exam questions here are mostly conceptual — name a CNN's layers, contrast LSTM and GRU, explain positional encoding — plus one very examinable numeric item: counting a network's trainable parameters. The homework quizzes (h8, h9) drill these.
What this chapter covers
- 01CNNs for images: CONV + ReLU + POOL + FC layers; parameter sharing and local receptive fields beat a plain dense net
- 02Max-pooling vs average-pooling and their drawbacks
- 03Counting trainable parameters of a Dense layer: inputs·units + units biases (Flatten/pooling add none)
- 04RNNs for sequences and their vanishing gradients over time
- 05LSTM (forget/input/output gates plus a cell state) vs GRU (reset/update gates)
- 06Transformer: multi-head self-attention with Q/K/V matrices, concatenate then W_O
- 07Positional encoding injects word-order information; softmax selects the next token at inference
- 08Softmax pᵢ = e^(oᵢ)/Σ e^(oⱼ); categorical cross-entropy loss CCE = −Σ yⱼ·log(ŷⱼ)
Counting the trainable parameters of a network
- +1Flatten turns the 32×32 input into 32·32 = 1024 features and has no trainable parameters.
- +1Dense(128): each of the 128 units has 1024 weights plus 1 bias → 1024·128 + 128 = 131,200.
- +1Dense(64): 128·64 + 64 = 8,256; Dense(10): 64·10 + 10 = 650.
- +1Total trainable parameters = 131,200 + 8,256 + 650 = 140,106.
Key terms
- Convolutional neural network (CNN)
- A network for grid data (images) using CONV, ReLU, POOL and FC layers; parameter sharing and local receptive fields make it far more effective than a plain dense net on images.
- Pooling
- A downsampling layer (max or average) that shrinks feature maps and adds no trainable parameters; max-pooling keeps the strongest activation, average-pooling the mean.
- LSTM
- A recurrent unit with a cell state and forget/input/output gates that let it retain information over long sequences, mitigating the vanishing gradient of a plain RNN.
- Self-attention
- The transformer mechanism that relates every position to every other via query/key/value matrices; multi-head attention runs several in parallel and concatenates them.
- Positional encoding
- Extra signals added to token embeddings so a transformer — which has no recurrence — knows the order of the sequence.
- Softmax / cross-entropy
- Softmax pᵢ = e^(oᵢ)/Σ e^(oⱼ) turns outputs into a probability distribution; categorical cross-entropy CCE = −Σ yⱼ·log(ŷⱼ) is the matching loss for one-hot labels.
Deep Neural Networks: CNNs, RNNs & Transformers FAQ
Why are CNNs better than plain dense networks for images?
Because they exploit image structure. Convolutional filters share weights across the image (far fewer parameters than a fully connected layer) and each neuron sees only a local receptive field, so the network learns translation-tolerant local features. That makes CNNs more accurate and more efficient than a dense net on images like MNIST.
LSTM vs GRU — what's the difference?
Both are gated recurrent units that fix the plain RNN's vanishing-gradient problem over long sequences. An LSTM has three gates (forget, input, output) and a separate cell state; a GRU is simpler, with two gates (reset, update) and no separate cell state. The GRU has fewer parameters and is faster; the LSTM is more expressive.
How do I count the trainable parameters of a network?
Go layer by layer. A Dense layer has inputs·units weights plus units biases. Flatten and pooling layers add nothing. Sum across layers. Watch the input size to the first Dense layer — it is the flattened feature count (e.g. 32×32 = 1024), and biases equal the number of units.
What does positional encoding do in a transformer?
Self-attention alone is order-agnostic — it treats the input as a set — so the model would not know word order. Positional encoding adds a position-dependent signal to each token's embedding, injecting sequence-order information so the transformer can distinguish 'dog bites man' from 'man bites dog'.
Exam move
Split deep learning into 'name it' and 'count it'. For the conceptual parts, build a one-page comparison: CNN layers and why they beat dense nets on images; max- vs average-pooling drawbacks; LSTM vs GRU gates; the transformer's self-attention plus positional encoding and softmax decoding. For the numeric part, drill Dense-layer parameter counting (inputs·units + units biases, Flatten/pooling add none) on a couple of architectures until it is quick and error-free. When a parameter count doesn't add up, ask Sia to recount it layer by layer with you.
Working through Deep Neural Networks: CNNs, RNNs & Transformers in COMP4318? Sia is AskSia’s AI Machine Learning tutor — ask any COMP4318 Deep Neural Networks: CNNs, RNNs & Transformers question and get a clear, step-by-step explanation grounded in how COMP4318 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
