COMP4318 · Machine Learning and Data Mining
Neural Networks: Perceptrons & Backpropagation
Week 7 builds neural networks from the perceptron up: activation functions, the multi-layer perceptron, gradient descent and the backpropagation algorithm that trains the weights. The exam typically asks a small MLP forward pass with sigmoids, or to name activation functions from their plots and say which suits a CNN and why. The homework quiz (h7) drills the forward pass and the update rule.
What this chapter covers
- 01Neuron output a = f(w·x + b), where f is the activation/transfer function
- 02Perceptron: step activation, binary output; learning rule e = t − a, w ← w + e·x, b ← b + e
- 03Perceptron limits: solves AND/OR/NAND but NOT XOR (needs a 2-layer net); converges iff data are linearly separable
- 04Multi-layer perceptron: feedforward, fully connected, differentiable activation (sigmoid f(z) = 1/(1+e^(−z)))
- 05Sigmoid derivative f'(x) = f(x)(1 − f(x)); the vanishing-gradient problem; ReLU y = max(0, x) as a fix
- 06Backpropagation: forward pass, then propagate error backward by gradient descent; Δw = η·δ·o
- 07Output-neuron δ = (t − o)·o·(1 − o); hidden-neuron δ = o(1 − o)·Σ w·δ (for a sigmoid)
- 08Gradient-descent variants: batch, stochastic (SGD) and mini-batch; the learning rate η as a hyperparameter
A forward pass through a small sigmoid network
- +1Hidden pre-activation: z_h = w1·x1 + w2·x2 + b_h = 0.5·1 + 0.5·2 − 1 = 0.5.
- +1Hidden activation: o_h = σ(0.5) = 1/(1 + e^(−0.5)) = 0.622.
- +1Output pre-activation: z_o = w·o_h + b_o = 1·0.622 + 0 = 0.622.
- +1Output activation: o = σ(0.622) = 1/(1 + e^(−0.622)) = 0.651.
Key terms
- Perceptron
- The simplest neural unit: a step activation on w·x + b giving a binary output; trained by e = t − a, w ← w + e·x. Converges only if the data are linearly separable and cannot learn XOR.
- Activation function
- The non-linear function f applied to a neuron's weighted sum; examples are step, sigmoid, tanh and ReLU.
- Sigmoid
- f(z) = 1/(1 + e^(−z)), a smooth (0,1) squashing function with derivative f'(z) = f(z)(1 − f(z)); the classic differentiable activation for MLPs.
- Backpropagation
- Training by a forward pass then a backward pass that propagates the error using gradient descent; weight update Δw = η·δ·o with output δ = (t−o)·o(1−o).
- Gradient descent (SGD)
- Iteratively updating weights against the loss gradient; batch uses all examples, stochastic (SGD) updates per example, mini-batch uses small groups.
- Learning rate η
- The step-size hyperparameter of gradient descent; too small learns slowly, too large oscillates or diverges.
Neural Networks: Perceptrons & Backpropagation FAQ
Why can't a single perceptron learn XOR?
Because a perceptron draws a single straight decision boundary, and XOR is not linearly separable — no straight line separates its two classes. It can learn AND, OR and NAND, which are linearly separable, but XOR needs a two-layer network (a hidden layer) to bend the boundary.
What's the difference between batch, mini-batch and stochastic gradient descent?
They differ in how many examples contribute to each weight update. Batch gradient descent sums the error over all examples then updates once per pass (smooth but slow). Stochastic gradient descent updates after every single example (noisy, zig-zag, but fast and can escape shallow minima). Mini-batch updates on small groups — the common compromise.
What is the vanishing-gradient problem?
In a deep network trained with sigmoids, the backpropagated error is repeatedly multiplied by the sigmoid derivative o(1−o), which is at most 0.25 and near 0 when the neuron saturates. Multiplying many such factors shrinks the gradient toward zero, so the earliest layers learn extremely slowly. ReLU (y = max(0,x)), which has gradient 1 for positive inputs, mitigates it.
How is the neural-network question examined?
Commonly as a small forward pass — compute each layer's weighted sum, apply the sigmoid, propagate to the output — and as a conceptual part naming activation functions from plots and saying which suits a CNN (ReLU) and why. Show every pre-activation and activation to earn the method marks.
Exam move
Rehearse the forward pass until it is mechanical: for each neuron compute w·x + b, apply the activation, feed forward. Memorise the sigmoid and its derivative f'(x) = f(x)(1 − f(x)), and the backprop deltas — output δ = (t − o)·o(1 − o) and hidden δ = o(1 − o)·Σwδ. Be ready to explain the perceptron's XOR limitation, the batch/mini-batch/SGD distinction, and the vanishing-gradient problem with ReLU as the fix. When the arithmetic of a forward pass slips, ask Sia to recompute it with fresh weights and check each activation.
Working through Neural Networks: Perceptrons & Backpropagation in COMP4318? Sia is AskSia’s AI Machine Learning tutor — ask any COMP4318 Neural Networks: Perceptrons & Backpropagation question and get a clear, step-by-step explanation grounded in how COMP4318 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
