ECE5179 Chap.12 Vision Transformers, Evaluation and Decoding
Vision Transformers, Evaluation and Decoding
Week 12 applies transformers to images and asks how language models are measured and used. The Vision Transformer cuts an image into patch tokens with a classification token and learns positions, outperforming convolutional networks only with very large data sets.
Perplexity and BLEU quantify a language model and its outputs, and decoding rules such as greedy search, temperature, top-k, top-p and beam search turn next-token probabilities into text.
What this chapter covers
- 01
Patch tokens, the classification token and learned positions
- 02
ViT variants and the data-scale comparison with ResNet
- 03
Linear probing and imageGPT
- 04
Perplexity as exponentiated cross-entropy
- 05
BLEU: clipped n-gram precision and brevity penalty
- 06
Greedy decoding, temperature, top-k and top-p
- 07
Beam search and choosing a decoder
Worked example · free
Greedy decoding against a beam of width two
- 1Greedy takes 'a' (0.5), then 'dog' (0.6), giving joint probability 0.30.
- 1Beam search keeps 'a' and 'the' after step one.
- 1Expansions score a dog 0.30, a cat 0.15, the cat 0.32 and the dog 0.06.
- 1The best two beams are 'the cat' and 'a dog'; the winner is 'the cat' at 0.32.
Key terms
- Patch embedding
- A patch embedding maps each flattened image patch to a token vector, usually with a convolution whose stride equals the patch size.
- Perplexity
- Perplexity is the exponential of the average negative log-likelihood a language model assigns to the true next tokens.
- Brevity penalty
- The brevity penalty scales BLEU down when a candidate sentence is shorter than its reference.
- Top-p sampling
- Top-p sampling draws the next token from the smallest set of tokens whose probabilities add up to at least p.
- Beam search
- Beam search keeps the k most probable partial sequences at each step and expands only those.
Vision Transformers, Evaluation and Decoding FAQ
How many tokens does a Vision Transformer read?
For an H by W image and patch size P there are (H/P) times (W/P) patch tokens, plus one classification token. A 224 by 224 image with 16 by 16 patches gives 196 patches and 197 tokens. Halving the patch size quadruples the patches and multiplies the attention table roughly sixteen-fold.
When does a Vision Transformer beat a convolutional network?
In the comparison the lectures present, ViTs trail ResNets when trained on ImageNet alone, draw level with ImageNet-21k, and pull ahead only with JFT-300M. Lacking convolution's built-in locality, a transformer needs much more data to learn it, so pre-trained ViTs are the practical choice for small data sets.
What does a perplexity score mean?
Perplexity is the exponential of the average cross-entropy on the true next tokens, so lower is better and a value of k behaves like choosing among k equally likely words. Scores depend on the data and tokeniser, so values from different data sets cannot be compared directly.
How does temperature change the next-token distribution?
Temperature divides the logits before the softmax. A value below one sharpens the distribution towards the most likely token, approaching greedy decoding, while a value above one flattens it towards uniform and increases variety. For logits 2, 1 and 0, temperature 0.5 gives about 0.87 to the top token and temperature 2 about 0.51.
Why does BLEU need a brevity penalty?
BLEU is built from precisions, which only ask whether the candidate's n-grams appear in the reference. A very short candidate made of safe words can therefore score perfect precision. The brevity penalty multiplies the score by exp(1 − r/c) when the candidate length c is below the reference length r.
What is the classification token in a Vision Transformer?
It is a learnable vector prepended to the sequence of patch tokens. Because it attends to every patch in every layer, its final vector summarises the whole image, and the classifier head reads only that vector to produce the class scores.
Exam move
Count ViT tokens for several image and patch sizes and relate each change to attention cost. Compute perplexity from a short list of token probabilities in both the average-log and product forms. Score one BLEU example fully, listing matched n-grams, clipping and the brevity penalty.
Then practise decoding by hand: temperature on three logits, top-k renormalisation, and a two-step beam search, and write one sentence for each on when you would choose it. Close the unit by linking back: perplexity is the exponential of the cross-entropy from the classification chapter, beam search is a search over the softmax outputs, and a ViT is the attention block from Chapter 10 applied to image patches.
Working through Vision Transformers, Evaluation and Decoding in ECE5179? Sia is AskSia’s AI Computer Science tutor — ask any ECE5179 Vision Transformers, Evaluation and Decoding question and get a clear, step-by-step explanation grounded in how ECE5179 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
