MATH7005 Chap.8 Scaled Dot-Product Attention
Scaled Dot-Product Attention
Addressing separated from content
Attention compares a query with key rows, turns those scores into a probability distribution and uses the weights to mix value rows. Keys decide relevance while values carry returned information.
Scaling by the square root of key width prevents large dot products from forcing a sharply saturated softmax.
What to verify
Check Q, K and V dimensions, normalise each query row across keys, apply masks before softmax and hand-calculate one weighted value sum.
The output width follows the values rather than the keys.
Rows carry independent questions
With several queries, each score row is normalised separately across the available keys. Batch rows and unrelated queries must never share probability mass.
Write the score-matrix dimensions before applying softmax and check that every permitted row sums to one after masking.
Scaling protects useful gradients
Larger key width tends to produce higher-variance dot products. Without the square-root divisor, softmax can become sharply peaked and leave very small gradients for most positions.
Apply scaling to query-key scores before normalisation, not to the values or to the completed weighted sum.
Interpretability needs intervention
An attention heatmap shows internal routing for a stated layer and head. It is not automatically a causal explanation. Replace or mask content associated with a high weight and measure the output change under fixed evaluation settings.
If the result barely moves, the visual weight did not establish the stronger claim.
What this chapter covers
- 01
Query-key dot product
- 02
Score scaling
- 03
Rowwise softmax
- 04
Weighted value sum
- 05
Attention masking
- 06
Self and cross-attention
Worked example · free
Mix two scalar values by attention
- 3Apply softmax to the scores.
- 2Multiply values by their weights.
- 2Add the contributions.
Key terms
- Query Vector
- The representation that asks which key rows are relevant.
- Key Vector
- The representation compared with a query to create a relevance score.
- Value Vector
- The content mixed using attention weights after relevance is calculated.
- Attention Matrix
- The rowwise probability distributions connecting queries to key positions.
- Scaling Factor
- The square-root key-width divisor that stabilises dot-product magnitude.
Scaled Dot-Product Attention FAQ
Why are attention scores scaled?
Dot-product variance grows with key dimension, making softmax excessively peaked and leaving weak gradients at most positions. Dividing by the square root of key width keeps the score scale more stable.
Why do keys and values need equal row counts?
Every key position receives one attention weight, and that weight must have a corresponding value row to mix. Their feature widths may differ, but their position counts must align.
Are attention weights explanations?
They are evidence about internal routing, not automatic causal explanations of a prediction. A stronger interpretability claim needs tests showing that changing the attended content changes the result in the asserted way.
Where should an attention mask be applied?
Apply it to disallowed scores before softmax so their exponentials contribute zero and permitted cells renormalise to one. Zeroing weights afterward changes the row scale and no longer performs the intended probability normalisation.
How can I hand-check batched attention code?
Remove batch and head axes for one small query-key-value example, reproduce the result by hand and then restore those axes. Assert that the same row remains unchanged, which isolates attention arithmetic from reshape and broadcasting errors.
What does the attention output dimension follow?
It keeps one row per query and the feature width of the values. Keys determine the number of scored positions and their similarity space, but the weighted content returned by the operation comes from V.
How can I detect a wrong softmax axis?
Check that every query row sums to one across key positions and compare one row with a hand calculation. Normalising across queries or batches may still produce plausible numbers, but it mixes independent predictions and violates the attention definition.
Assessment move
Choose one query, three keys and simple value vectors. Calculate dot products, divide by the square root of key width, apply stable softmax and form the weighted value sum. Check that the result lies within the coordinate range of the values before later projections. Add a mask to one key before normalisation and confirm that its weight becomes zero while the permitted cells renormalise.
Extend the example to two queries and label the rowwise axis. In code, assert Q, K, V, score and output shapes and compare one row with the hand result. For interpretation, label the layer and head and perform a content-removal test instead of treating colour intensity as proof.
Working through Scaled Dot-Product Attention in MATH7005? Sia is AskSia’s AI Computer Science tutor — ask any MATH7005 Scaled Dot-Product Attention question and get a clear, step-by-step explanation grounded in how MATH7005 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
