FIT1043 · Introduction to Data Science
Regression Analysis
Week 6 of Monash FIT1043 Introduction to Data Science deepens regression into the core modelling trade-off of the unit: bias vs variance, and its symptoms overfitting and underfitting. Using polynomial fits of increasing order, it shows how a too-simple model has large bias and a too-complex model has large variance and 'goes wild', with a middle order fitting best. It introduces the No Free Lunch idea and ensemble methods at an introductory level. Bias vs variance is a recurring short-answer topic, so getting the definitions and the trade-off direction right is high-yield for the final.
What this chapter covers
- 01Parameter estimation: fitting model coefficients, e.g. by least squares (best fit = least squared error over the x-range)
- 02Model family and hyper-parameters: a family is a class of models set by hyper-parameters (here the polynomial ORDER); a member is fixed by its coefficients
- 03Bias: how closely a member of the family CAN fit the truth — simple/low-order = large bias, complex/high-order = small bias
- 04Variance: the spread between fitted curves across samples vs the best possible (infinite-data) fit
- 05The bias-variance trade-off: underfit (too simple) vs overfit (too complex, erratic on finite data)
- 06Overfitting vs underfitting: a high-order poly with about as many coefficients as data points 'goes wild'
- 07No Free Lunch theorem: no single model is best for every problem
- 08Ensemble models: combining models to improve on any single one (introductory)
Diagnose underfitting vs overfitting across polynomial orders
- +1Order 3 = UNDERFIT: too simple to capture a curved truth, so it has LARGE BIAS (a member of this low-order family cannot fit the truth well) but small variance (fits from different samples look similar).
- +1Order 12 = GOOD FIT: complex enough to follow the true shape without chasing noise. Order 25 = OVERFIT: with about as many coefficients as data points it has small bias but LARGE VARIANCE — it 'goes wild', fitting the noise and swinging between points, so it generalises poorly.
Key terms
- Bias
- How closely a member of a model family CAN fit the truth. Simple/low-order models have large bias (they cannot capture a complex truth); complex/high-order models have small bias.
- Variance
- The spread between the curves a model fits on different samples, measured against the best-possible fit from infinite data. Simple models have small variance; complex models can have large variance when they 'go wild'.
- Overfitting
- When a too-complex model (e.g. a high-order polynomial with about as many coefficients as data points) fits the noise as well as the signal, giving small training error but poor generalisation.
- Underfitting
- When a too-simple model cannot capture the true pattern, leaving large systematic error (large bias) even on the training data.
- Bias-variance trade-off
- The tension whereby reducing bias (more complex model) tends to raise variance and vice versa; the best model balances the two rather than minimising either alone.
- No Free Lunch theorem
- The principle that no single model or algorithm is best across all possible problems — model choice must suit the specific task.
Regression Analysis FAQ
What is the difference between bias and variance?
Bias is how well a model CAN fit the truth: a simple, low-order model has large bias because it is too rigid to capture a complex pattern. Variance is how much the fitted model changes across different data samples: a complex, high-order model can have large variance because it chases the noise and 'goes wild'. Simple models are large-bias/small-variance; complex models are small-bias/large-variance.
How do I tell overfitting from underfitting?
Underfitting is a too-simple model that misses the pattern (large bias) even on the training data. Overfitting is a too-complex model that fits the training data — including its noise — almost perfectly but generalises badly (large variance). In the unit's polynomial lab, order 3 underfits, order 25 overfits, and a middle order around 12 is the good fit.
Does a higher-order polynomial always fit better?
No. It fits the TRAINING points better (small bias), but past a point it starts fitting the noise and swings wildly between points (large variance), so its predictions on new data get worse. This is exactly the overfitting failure — the best choice balances bias and variance rather than maximising flexibility.
What is the No Free Lunch theorem?
It is the idea that there is no single model or algorithm that is best for every problem — a model that excels on one task can be poor on another. The practical takeaway is that you must choose and evaluate models for the specific problem rather than expecting one 'best' method to always win.
Can AI help me with bias-variance in FIT1043?
Yes. Sia can explain the bias-variance trade-off, classify polynomial fits as under/over/good, and quiz you on the direction (simple = large bias, complex = large variance), step by step. It explains the concepts and checks your reasoning; it does not complete graded work for you, and Monash academic-integrity rules apply. Confirm details on Moodle.
Exam move
Bias vs variance is the single most examinable idea this week, so memorise the direction until it is reflexive: simple/low-order = large bias, small variance, underfit; complex/high-order = small bias, large variance, overfit; the winner is the balanced middle. Rehearse the polynomial-order story (order 3 vs 12 vs 25) as a two-line short answer that names the fit AND the bias/variance behaviour. Add crisp one-liners for the No Free Lunch theorem and what an ensemble is. Because the exam rewards justified definitions, always attach the 'why' to your classification rather than just labelling a fit.
Working through Regression Analysis in FIT1043? Sia is AskSia’s AI Information Technology tutor — ask any FIT1043 Regression Analysis question and get a clear, step-by-step explanation grounded in how FIT1043 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
