QBUS2820 Chap.5 Prediction Error, Bias and Variance
Prediction Error, Bias and Variance
Predictive quality is defined on future cases, not on the observations used to fit a model. This chapter separates training, validation and test roles, develops expected test error, and decomposes squared prediction error into bias, variance and irreducible noise.
It explains why flexibility can reduce bias while increasing sample sensitivity, why the test-error curve can be U-shaped, and why data leakage through preprocessing or repeated test inspection produces optimistic evidence. Expected error is tied to a future data-generating setting. A low value under random folds may not transfer to a later period, another store or a new customer if the split does not imitate that setting.
Sampling variation also affects the estimated error itself, so fold-level results and repeated resampling help distinguish a robust advantage from a lucky partition. Hyperparameter searches create selection pressure: the more candidates inspected, the easier it is to overfit validation noise. Nested evaluation or a final untouched test set protects the last performance claim.
Preprocessing must be treated as model fitting, including imputation, scaling, feature screening and threshold choice. Grouped or chronological splits are often required when rows share an entity or time structure. The irreducible-error label should not become an excuse for unexplained residuals; it refers to variation left after using the relevant predictors and model target, and new information can change that boundary.
Monitoring after deployment tests whether the evaluation assumptions continue to hold.
What this chapter covers
- 01
Training and test error
- 02
Expected prediction error
- 03
Bias squared
- 04
Prediction variance
- 05
Irreducible error
- 06
Underfitting and overfitting
- 07
Validation boundaries
- 08
Leakage and deployment matching
Worked example · free
Compare two error decompositions
- +1The marks used here structure this original exercise and are not an official University marking allocation. Square Model A bias: 3² = 9.
- +1Add Model A components: 9 + 4 + 9 = 22.
- +1Square Model B bias: 1² = 1.
- +1Add Model B components: 1 + 15 + 9 = 25.
- +1Prefer A at this input because its variance saving exceeds B's bias reduction.
Key terms
- Training Error
- Loss measured on cases used to estimate the model.
- Test Error
- Loss measured on independent cases not used for model choices.
- Expected Error
- Average future loss over relevant cases and potential fitted samples.
- Bias
- Systematic difference between the average fitted prediction and the true target.
- Variance
- Prediction changes caused by using different training samples.
- Data Leakage
- Evaluation information influencing fitting, preprocessing, selection or tuning.
- Validation Noise
- Sampling variation in an estimated validation loss caused by the particular cases assigned to each held-out split.
Prediction Error, Bias and Variance FAQ
Why can test error rise with flexibility?
A more flexible model can follow noise in its training sample. Bias may fall, but prediction variance can rise enough to increase future loss, creating a U-shaped test-error pattern.
What belongs inside cross-validation?
Every data-dependent step belongs inside each training fold, including imputation, scaling, transformation, feature screening and tuning. Applying those steps globally leaks information from held-out cases.
How should time-dependent data be split?
Use a design that respects the decision clock, often training on earlier periods and validating on later periods. Random mixing can expose future conditions and exaggerate expected deployment performance.
Why should cross-validation be repeated?
Different fold assignments can change an estimated loss, especially in small or heterogeneous datasets. Repetition reveals that variation and shows whether a model advantage persists beyond one convenient partition.
What is nested evaluation for?
An inner resampling loop chooses models or tuning values, while an outer loop estimates performance after that choice. This separates selection from evaluation when a single final test set is unavailable.
Exam move
Memorise the squared-error decomposition by deriving its meaning, not by chanting labels. Draw three curves against flexibility: bias squared falling, variance rising and test error reaching an interior minimum. Use original numeric triples to compute totals and explain why irreducible noise limits every candidate equally. Build a data-role table that states what training, validation and test samples may influence.
Then audit common leakage routes: scaling on all cases, selecting features before cross-validation, tuning a threshold on test outcomes, and repeatedly revising after test inspection. For each source, write the safe alternative. Practise choosing split units that match deployment, including chronological splits and grouped customer splits.
End with a model-comparison paragraph that reports loss, fold variation, stability, interpretability and operational constraints instead of declaring a winner from one lucky validation number.
Working through Prediction Error, Bias and Variance in QBUS2820? Sia is AskSia’s AI Statistics tutor — ask any QBUS2820 Prediction Error, Bias and Variance question and get a clear, step-by-step explanation grounded in how QBUS2820 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
