IS5542 Chap.1 How Generative AI Works: From Rules to Transformers
How Generative AI Works: From Rules to Transformers
Prediction is the common thread
The course begins with the difference between two kinds of machine learning task. Given a set of weather readings, asking whether it will rain is classification and asking how much is regression. A language model does something very close to the second: it estimates which token should come next, over and over, and a fluent paragraph is the result.
Keeping that picture in mind explains most of what follows in the course, including why models sound confident when they are wrong.
Three problems that language adds
Sentences vary in length, meaning depends on words far back in the text, and words must become numbers before a network can use them.
Tokenisation breaks text into units, embeddings map each unit to a few hundred numbers that keep related meanings close, and attention lets every token be weighed against every other.
The amount of text that can be weighed at once is the context window, which the lecture compares across leading models.
The history in four steps
Rule-based chatbots of the 1960s matched patterns; symbolic systems proved brittle on messy data and an AI winter followed; large datasets and GPU computing powered the deep learning turn from 2012; and the 2017 Transformer led to pre-trained generative models, ChatGPT's mainstream moment and, more recently, agents and multimodal models.
Protein-structure prediction appears as proof that the same techniques reach far beyond chat.
Why a manager needs this
Every business decision later in the course, from grounding answers in company documents to budgeting for context length, rests on knowing that the model predicts rather than looks things up.
What this chapter covers
- 01
Classification and regression in one weather example
- 02
Why natural language is a harder input
- 03
Tokens, sub-words and embeddings
- 04
Attention and the context window
- 05
What large, general and pre-trained mean
- 06
Multimodal input and output
- 07
From pattern matching to the Transformer
- 08
Hallucination as a prediction failure
Worked example · free
Explain a wrong answer without blaming the model
- 2Say what the model was doing when it produced the answer.
- 2Identify what was missing from its context.
- 2Propose the fix and how to check that it worked.
Key terms
- Tokenisation
- Splitting text into the units a model processes, which may be whole words, parts of words or characters.
- Word Embedding
- A short vector of real numbers representing a word or token, arranged so that related meanings sit close together.
- Multi-head Attention
- The mechanism that lets a model weigh each token against every other token in its input when predicting the next one.
- Pre-training
- The first, broad stage of training on general text, after which a model can be fine-tuned for narrower tasks.
- Multimodality
- The ability to take or produce more than one kind of data, such as text, image, audio and video.
- AI Winter
- A period of reduced progress and funding after symbolic systems failed to cope with ambiguous, unstructured data.
How Generative AI Works: From Rules to Transformers FAQ
Why does a language model sound confident even when it is wrong?
Because fluency comes from predicting likely word sequences, not from checking facts. A sentence can be highly probable given the model's training and still be false for your situation, which is why grounding answers in supplied documents matters more than choosing a model that writes well.
What does the context window mean for a business?
It limits how much material the model can consider at once. Larger windows allow whole contracts or long recordings in a single request, while smaller ones force the organisation to split documents or retrieve only relevant passages, which affects cost, speed and how easily answers stay faithful to the source.
Is generative AI the same thing as machine learning?
No. Machine learning is the broader approach of fitting weights to data; deep learning is one kind of it, and generative models are deep networks trained to produce new content rather than only label existing inputs, which is the shift the course describes from recognising a cat to drawing one.
Exam move
Draw the nested diagram of AI, machine learning and deep learning from memory, then add tokens, embeddings and attention as three arrows showing what each solves. If you can explain a hallucination using only those three words and the context window, you are ready for the multiple-choice history items as well.
Working through How Generative AI Works: From Rules to Transformers in IS5542? Sia is AskSia’s AI Information Technology tutor — ask any IS5542 How Generative AI Works: From Rules to Transformers question and get a clear, step-by-step explanation grounded in how IS5542 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
