MAIE5421 Chap.1 Machine Learning Foundations and the Learning Workflow
Machine Learning Foundations and the Learning Workflow
Two ways to make a machine behave intelligently
The course frames its subject with a choice. One route is to write the behaviour down yourself as explicit rules, which is quick for narrow tasks and brittle everywhere else, because the rules are often incomplete and need deep expertise to write.
The other route is to give the machine examples and let it find the regularities, which is what machine learning means here: training an algorithm on data so that it recognises patterns and makes decisions without being programmed task by task, and so that its performance improves as it sees more data.
Computer vision is the clearest case for the second route, since nobody can write down a rule that recognises a face under every lighting condition.
Artificial intelligence, machine learning and deep learning
The three terms are nested rather than competing.
Artificial intelligence covers any system that imitates human cognitive functions such as perception, problem solving and language, including expert systems that never learn. Machine learning is the subset that learns from data. Deep learning is the subset of machine learning that uses neural networks with many layers.
The lecture also traces the history from the Turing test and the perceptron in the 1950s, through decision trees, nearest neighbours, the AI winter, back-propagation and support vector machines, to the big-data era, deep learning and foundation models.
Tasks, samples and the way data is divided
Two task families recur through the course.
Regression predicts a continuous number such as a temperature, a price or a depth, and classification predicts one of a finite set of categories such as spam or not spam, or which digit is written. A sample is a pair made of an attribute vector and its target, and a dataset is a collection of such pairs.
It is divided into a training set to fit the model, a validation set to guide design choices while tuning, and a test set kept back until the end; the slides quote 80/10/10 and 90/5/5 as typical proportions.
Learning types are sorted by what the data carries: labels for supervised learning, none for unsupervised learning, a little for semi-supervised learning, and rewards from an environment for reinforcement learning.
The failure modes every model faces
A model can fail by memorising or by being too simple.
Overfitting shows as excellent training performance with poor results on new data, and the slides attribute it to a model that is too complex or a training set that is too small. Underfitting shows as poor results on both, from a model too simple for the data.
The practical skill is to read both numbers together: a large gap between training and validation error points one way, two high and similar errors point the other, and the repair differs accordingly.
What this chapter covers
- 01
Rule-based systems against data-driven learning
- 02
How artificial intelligence, machine learning and deep learning nest
- 03
A brief history from the perceptron to foundation models
- 04
Regression and classification as the two core tasks
- 05
Data samples, datasets and the training, validation and test split
- 06
Supervised, unsupervised, semi-supervised and reinforcement learning
- 07
Overfitting, underfitting and how to tell them apart
Worked example · free
Pick the model complexity from training and validation error curves
- 2The lowest validation error is 0.25 at level 3, so choose level 3.
- 1Level 1: training 0.38 and validation 0.38, both high and equal, so it underfits.
- 2Level 6: training 0.09 against validation 0.39, a gap of 0.30, so it overfits.
- 1Report the chosen model's score once on the held-out test set, never on the validation set used to choose it.
Key terms
- Rule-based System
- A system whose behaviour is fixed by rules that people write in advance, so it is knowledge-driven rather than learned from data.
- Deep Learning
- The subset of machine learning that uses neural networks with many layers, the depth that gives the method its name.
- Data Sample
- One pair made of an attribute vector describing an input and the class or regression target that goes with it.
- Test Set
- The portion of a dataset held back until training and tuning are finished, used once to estimate performance on unseen data.
- Semi-supervised Learning
- Learning that combines a small amount of labelled data with a large amount of unlabelled data during training.
- Reinforcement Learning
- Learning in which an agent takes actions and improves its decisions by optimising the cumulative reward those actions earn.
- Underfitting
- Failing to learn the relationship between input and target, so performance is poor on the training data and on new data alike.
Machine Learning Foundations and the Learning Workflow FAQ
Is face recognition a regression or a classification problem?
When the system must say which of a known set of people is in the picture, it is classification, because the output is one category from a finite list; the lecture lists face recognition among multi-class problems. Face verification, deciding whether two photographs show the same person, is a binary classification. Estimating a continuous attribute from a face, such as age in years, would be regression instead.
Why do we need a validation set as well as a test set?
Because tuning uses up the independence of whatever data it looks at. Each time you compare designs on a subset and keep the better one, the winning score is biased upward on that subset. The validation set absorbs that bias during development, which leaves the test set untouched for one honest estimate at the end of the project.
How can I tell overfitting from underfitting?
Compare training and validation error side by side. A small training error with a much larger validation error means the model has learned its training data too closely, which is overfitting. Two high errors that sit close together mean the model is too simple to capture the relationship, which is underfitting. The size of the gap, not either error alone, carries the diagnosis.
Assessment move
Write the definition of machine learning from memory in one sentence, then list three vision tasks and label each as regression or classification and as supervised or unsupervised. If any label takes you more than a few seconds, reread that part of the chapter, because every later method assumes these distinctions are automatic.
Working through Machine Learning Foundations and the Learning Workflow in MAIE5421? Sia is AskSia’s AI Computer Science tutor — ask any MAIE5421 Machine Learning Foundations and the Learning Workflow question and get a clear, step-by-step explanation grounded in how MAIE5421 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.