ECE5179 Chap.11 Tokenisation, Positional Encoding and BERT
Tokenisation, Positional Encoding and BERT
Week 11 builds the pieces that turn the transformer into a language model. Tokenisation cuts text into units, with subword schemes such as byte-pair encoding balancing vocabulary size against sequence length. Positional encodings write word order into the input, because attention alone cannot see it.
BERT then shows how an encoder is pre-trained by masked language modelling and fine-tuned with small heads for classification, entity tagging and question answering.
What this chapter covers
- 01
Character, word and subword tokens
- 02
Byte-pair encoding merge by merge
- 03
Permutation equivariance and the need for position
- 04
Properties of an ideal positional encoding
- 05
Sinusoidal, learned and rotary encodings
- 06
BERT pre-training by masked language modelling
- 07
Fine-tuning heads for classification, tagging and question answering
- 08
Encoder models versus left-to-right language models
Worked example · free
Sinusoidal positional codes for two positions
- 1With d = 4 there are two frequencies: the divisor is 10000^0 = 1 for dimensions 0 and 1, and 10000^(2/4) = 100 for dimensions 2 and 3.
- 1Position 1: (sin 1, cos 1, sin 0.01, cos 0.01) = (0.8415, 0.5403, 0.0100, 1.0000).
- 1Position 2: (sin 2, cos 2, sin 0.02, cos 0.02) = (0.9093, -0.4161, 0.0200, 0.9998).
- 1The fast pair (dimensions 0 and 1) changes a lot between neighbours, while the slow pair barely moves and only separates positions that are far apart.
Key terms
- Byte-pair encoding
- Byte-pair encoding builds a subword vocabulary by repeatedly merging the most frequent adjacent pair of symbols.
- Positional encoding
- A positional encoding is a vector added to each token embedding to record the token's position in the sequence.
- Masked language modelling
- Masked language modelling trains an encoder to recover tokens that have been hidden or replaced in its input.
- Fine-tuning
- Fine-tuning continues training a pre-trained model, with a small task head, on a specific labelled task.
Tokenisation, Positional Encoding and BERT FAQ
Why do transformers need positional encodings?
Dot-product attention is permutation equivariant: shuffling the input tokens simply shuffles the outputs in the same way. Two sentences with the same words in a different order would therefore look identical. Adding a position vector to each token makes order part of the input, so attention can use it.
What makes a good positional encoding?
The lectures list three properties: a unique code for every position, values that stay bounded, and the ability to generalise to sequences longer than those seen in training. Sinusoids of geometrically spaced frequencies meet all three, while plain integers grow without bound and fractions of the sequence length are not unique.
How does BERT differ from a left-to-right language model?
BERT is an encoder trained to fill in masked tokens using words on both sides, which suits understanding tasks such as classification and extraction. A left-to-right model predicts each token from the earlier ones only, which makes it natural for generating text but gives it less context for each prediction.
Why can a learned positional encoding not handle longer sequences?
A learned encoding stores one trainable vector per position up to the longest training length. Beyond that length there is simply no learned vector to use. A fixed formula such as the sinusoidal encoding defines a code for every position, and rotary encodings build relative position into the attention score itself.
How large are BERT-base and BERT-large?
BERT-base has 12 layers, a width of 768 and 12 attention heads, about 110 million parameters. BERT-large has 24 layers, a width of 1024 and 16 heads, about 340 million parameters. Both were pre-trained on BooksCorpus and English Wikipedia with a vocabulary of roughly 30,000 tokens.
Exam move
Run byte-pair encoding by hand on a tiny corpus of your own, recounting pairs after every merge, then tokenise an unseen word by replaying the merges in order. Compute sinusoidal encodings for two positions with a small model width and check them against the three properties.
For BERT, describe the masking recipe exactly as the lecture slides give it and match each downstream task to its head, stating which output vectors it reads and which loss it uses. When you meet a new tokeniser in practice, predict its splits for a long word, a rare word and a line of code before running it, then compare; the mismatches show which merges it learned from its training data.
Working through Tokenisation, Positional Encoding and BERT in ECE5179? Sia is AskSia’s AI Computer Science tutor — ask any ECE5179 Tokenisation, Positional Encoding and BERT question and get a clear, step-by-step explanation grounded in how ECE5179 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
