MATH7005 Chap.10 Efficient Transformers and Distillation
Efficient Transformers and Distillation
Different methods remove different costs
KV caching avoids recomputing history during autoregressive decoding, rotary position methods change query-key geometry, online attention reduces stored intermediate work and distillation transfers behaviour into a smaller model.
The chapter requires every efficiency claim to name the resource saved and the result retained.
What to verify
Benchmark methods under the same workload and hardware assumptions, record cache growth, compare exact attention outputs where appropriate and separate token-level soft targets from sentence-level teacher-generated targets.
Caching exchanges computation for memory
Past keys and values stay fixed during autoregressive generation, so storing them avoids repeated projections.
The cache grows with sequence length, layer count, head count and head width, and both tensor families must be counted. Measure latency over realistic context lengths because the first decoding step cannot reveal the later saving.
Execution changes can preserve the function
Online softmax maintains a running maximum and rescales earlier contributions whenever that maximum changes.
Online attention extends the recurrence to the value accumulator, reducing materialised intermediates while targeting the same result. Compare with a reference output under a stated tolerance before making a speed claim.
Distillation changes the supervision
Token-level training preserves the teacher distribution at each position, including plausible alternatives.
Sentence-level training uses complete teacher-generated sequences as pseudo-targets after an offline decoding pass. Compare model size, latency and task failures as well as an average score, because a smaller student may lose behaviour important to the product user.
What this chapter covers
- 01
KV caching
- 02
Rotary position
- 03
Stable online softmax
- 04
Online attention
- 05
Token distillation
- 06
Sentence distillation
Worked example · free
Count one layer's key-value cache
- 2Count the key values.
- 2Count the value values.
- 2Combine the two stores.
Key terms
- KV Cache
- Stored past keys and values reused during autoregressive generation.
- Rotary Embedding
- A position-dependent rotation applied to paired query and key coordinates.
- Online Softmax
- A stable recurrence maintaining a running maximum and rescaled denominator.
- Token Distillation
- Student training against the teacher probability distribution at each token position.
- Sentence Distillation
- Student training on complete sequences generated by the teacher model.
Efficient Transformers and Distillation FAQ
What work does a KV cache save?
It prevents the model from recomputing keys and values for earlier tokens at every decoding step. The new query still compares with the stored history, and the cache consumes memory that grows with sequence length.
Does online attention approximate ordinary attention?
The mathematical reorganisation can produce the same attention result while avoiding full score-matrix materialisation, subject to normal floating-point differences. It changes execution and memory traffic rather than the intended function.
How do token and sentence distillation differ?
Token-level training preserves a soft distribution of alternatives at every position. Sentence-level training uses complete teacher-generated outputs as pseudo-targets, simplifying later training but discarding much of the teacher probability distribution.
How should an efficiency benchmark be reported?
Fix model, hardware, batch and sequence settings, include warm-up and repeated measurements, and report latency with peak memory. Name the baseline time and workload so a percentage improvement can be reproduced rather than read in isolation.
What result should online attention preserve?
It reorganises stable normalisation and value accumulation to reduce intermediate storage while targeting ordinary attention output. Compare a small case with a trusted reference under a stated floating-point tolerance before measuring speed.
Why report memory beside latency?
Caching and other execution choices may accelerate a workload while consuming more memory. A deployment claim is incomplete unless the benchmark states both resources at realistic sequence lengths and identifies any training or offline cost moved elsewhere.
Which costs belong in a student-model comparison?
Include teacher generation or logits used during training, student training, deployed latency, peak memory and model size. Separate one-off preparation from repeated inference so the claimed saving can be evaluated at the intended usage scale.
Assessment move
Build a table with one row for caching, rotary position, online attention, token distillation and sentence distillation. Name the problem, mathematical change, saved resource, added cost and retained behaviour for each. Calculate cache scalars for a small model and extend the count across layers. Work one running-maximum softmax update, making the rescaling of earlier terms explicit.
Compare a teacher probability vector with its argmax target and identify the information discarded. For benchmarks, fix hardware, batch, sequence length and model, include warm-up and repeated measurements, and report latency with peak memory. Inspect teacher-student disagreements that affect the intended product action rather than relying only on an overall average.
Working through Efficient Transformers and Distillation in MATH7005? Sia is AskSia’s AI Computer Science tutor — ask any MATH7005 Efficient Transformers and Distillation question and get a clear, step-by-step explanation grounded in how MATH7005 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
