ECE5179 Chap.10 Attention and the Transformer Block
Attention and the Transformer Block
Weeks 10 and 11 introduce attention, the operation that lets each token build its own vector from all the others. Queries, keys and values are projections of the token matrix, the scaled softmax of their dot products gives each token a budget of weights, and the output is a weighted average of the values.
The chapter works an attention row by hand, explains multi-head attention, Layer Norm and residual adds, and the quadratic cost in sequence length.
What this chapter covers
- 01
Why one word needs context-dependent vectors
- 02
Queries, keys and values as learned projections
- 03
Scaled dot-product attention and the √d_k divisor
- 04
Attention weights as convex combinations
- 05
Multi-head attention and per-head width
- 06
The transformer block: attention, MLP, residuals and Layer Norm
- 07
Computational cost and permutation equivariance
Worked example · free
Compute one row of an attention table
- 1Dot products of the first query with the three keys are 1, 0 and 1.
- 1Dividing by √2 gives scaled scores 0.707, 0 and 0.707.
- 1Exponentials 2.028, 1 and 2.028 sum to 5.056, so the weights are 0.401, 0.198 and 0.401.
- 1The output is 0.401(1, 2) + 0.198(0, 1) + 0.401(3, 0).
- 1Adding the terms gives (1.604, 1.000).
Key terms
- Query vector
- A query vector is the projection of a token that is compared against every key to decide where it attends.
- Value vector
- A value vector is the projection of a token that is averaged, with attention weights, to form the output.
- Multi-head attention
- Multi-head attention runs several attention operations with separate projections in parallel and concatenates their outputs.
- Layer normalisation
- Layer normalisation standardises each token across its own features, without using batch statistics.
Attention and the Transformer Block FAQ
Why divide the attention scores by the square root of d_k?
If query and key entries are roughly independent with unit variance, their dot product has variance close to d_k. Large scores push softmax towards a one-hot output whose gradients nearly vanish. Dividing by √d_k restores unit scale, which the lectures describe as scaling the softmax for stability.
Why use separate query and key projections?
Scoring tokens with their raw dot products produces a symmetric matrix, so token A would attend to token B exactly as strongly as B attends to A. Language relationships are often one-sided, such as apple pointing to fruit more than the reverse, and separate learned projections allow that asymmetry.
Why does attention cost grow quadratically with sequence length?
Every token's query is compared with every token's key, producing an n by n table of scores, and that table then multiplies the values. Both steps scale with n squared times d, so doubling the context roughly quadruples the attention work, while the projections and MLP grow only linearly in n.
What does multi-head attention add over a single head?
A single head must either average several relationships or pick one and lose the others. Several heads with separate projections can each specialise, for instance one tracking syntax and another tracking topic, and their outputs are concatenated and projected back to the model width. With per-head width d/H the parameter count stays the same.
What is the difference between Pre-LN and Post-LN transformer blocks?
Post-LN applies Layer Normalisation after each residual addition, the original arrangement, which tends to need a learning-rate warm-up to train stably. Pre-LN normalises the input of each sublayer before attention or the MLP and leaves the residual path untouched, which the lectures present as easier to optimise.
What does the MLP inside a transformer block do?
Attention mixes information across tokens but applies no non-linearity to each token's features beyond the softmax weighting. The row-wise MLP, usually with a hidden width about four times the model width, transforms each token's features independently, giving the block the per-token processing that attention alone lacks.
Exam move
Practise one full attention row by hand with small vectors, writing raw scores, scaled scores, exponentials, weights and output on separate lines, and check that the weights sum to one. Then track shapes for a whole layer: X, Q, K, the score table, the output and the multi-head concatenation. Use published model sizes to compute per-head widths.
Finish by explaining permutation equivariance in two sentences, because it sets up the need for positional encodings. Add Layer Norm to your shape practice: compute the mean and standard deviation of one token's features, normalise it, and compare with batch normalisation, which would instead use one feature across many tokens or samples.
Working through Attention and the Transformer Block in ECE5179? Sia is AskSia’s AI Computer Science tutor — ask any ECE5179 Attention and the Transformer Block question and get a clear, step-by-step explanation grounded in how ECE5179 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
