BMS5010 Chap.4 Measuring Error in a Regression Model
Measuring Error in a Regression Model
Three summaries of the same column of mistakes
A regression model produces a number and the outcome it predicted is a number, so a difference is available for every row. What is not obvious is how to summarise a column of differences into one figure, and the course gives three answers that can rank two models differently. Average absolute difference is described as the first-order loss.
Average squared difference is the second-order loss. The root of the second is the measure the course describes as easier to interpret, because it shares the units of the target variable.
Choosing among them is a modelling decision rather than an administrative one, and it is made before results are seen rather than after.
The whole technical difference is the exponent
Both of the first two measures average something over the same residuals. The absolute version treats a mistake of six as six times worse than a mistake of one; the squared version treats it as thirty-six times worse.
Everything else in the topic follows from that. The course states the consequence in one line each: absolute error is less sensitive to outliers and encourages sparse models in which many coefficients become zero, while squared error penalises large errors more heavily and is therefore more sensitive to outliers.
Those two lines also carry the correspondence to the two regularisation penalties, which is worth reading in both directions.
Sensitive is not the same as wrong
Neither choice is the safe one. Sensitivity to outliers is a defect when the extreme values are recording errors, because the model then spends its capacity fitting mistakes.
It is a feature when the extreme values are the patients you most need to get right, which in clinical work is usually the case. So the question is never which measure is more robust in general.
It is whether the largest residuals in your own data are noise or signal, and you answer that by sorting the residual column and looking at the largest few rather than by choosing a measure in advance from a textbook.
Units, and the only reason the third measure exists
Squared error is expressed in the square of whatever the outcome was measured in, so a figure of several hundred on a length-of-stay model is several hundred squared days, which nobody can picture.
Taking the square root restores the units. Because the square root is a monotone operation it can never reorder two models, so there is no argument to be had about whether to compare on the squared figure or its root; the root buys interpretation and not information.
There is a real argument about whether to compare on squared or absolute error, because that pair genuinely can disagree, and keeping those two questions apart avoids a common muddle.
What a single summary figure hides
A summary conceals which rows produced it, and the two measures conceal different things.
A model that is uniformly mediocre and one that is excellent with a handful of catastrophes can report the same absolute error and very different squared error, and those are different problems with different repairs. Sorting the residuals and inspecting the largest five takes a minute and answers the question the summary cannot.
An assessor asking why you chose your error measure is, in most cases, asking whether you did this, so having the answer ready converts an open question into a short one.
The control the assignment asks for, and why it is there
The individual assignment asks for three squared-error figures on a fitted model: one on the training data, one on held-back data, and one on random inputs. The third is not decoration.
A model with no real relationship in it still produces an impressive-looking training figure, and scoring it on inputs carrying no information at all is the cheapest way to see that. If the held-back figure is not far below the random-input figure, the model has learned nothing worth reporting whatever its training error says.
A defensible comparison table then has one row per model, both figures side by side, the root of the held-back figure in the units of the outcome, and one sentence naming which model you would use.
What this chapter covers
- 01
Average absolute difference, average squared difference and the root of the second
- 02
How the exponent alone produces every other difference between them
- 03
Outlier sensitivity as a property of your data rather than of the measure
- 04
The correspondence between each loss and its regularisation penalty
- 05
Why taking the root cannot reorder two models
- 06
Reading the residual column instead of only its summary
- 07
The random-input control, and the shape of a defensible comparison table
Rank two models whose absolute errors are identical
- 2Compute the average absolute error for each model.
- 3Compute the average squared error and its root for each.
- 3Choose a model for this user and justify the choice in one sentence.
Key terms
- Residual
- The difference between a model's prediction and the observed outcome for a single row, which is the raw material every regression error measure summarises.
- Mean Absolute Error
- The average of the absolute residuals, which treats every unit of mistake alike and is expressed in the units of the outcome.
- Mean Squared Error
- The average of the squared residuals, which grows faster than the mistake itself and so weighs a few large misses far more heavily than many small ones.
- Monotone Transformation
- A change that preserves order, such as taking a square root, which is why converting squared error to its root can never reorder two models.
- Random-Input Control
- Scoring a fitted model on inputs carrying no information, which gives the error figure a genuinely uninformative model would produce on the same data.
Measuring Error in a Regression Model FAQ
Should I fit on one measure and report another?
Often yes, and it is worth saying so explicitly. Fit on the measure that matches the cost of being wrong, because that is what the optimisation will chase. Report the figure a reader can interpret, which usually means the root of squared error in the units of the outcome.
Reporting the absolute error alongside costs one line and answers a different question, namely how the model does on a typical patient rather than on its worst ones. What is not defensible is fitting on one measure and reporting only another without saying which was which.
A single outlier dominates my squared error. Should I remove it?
Only if you can say why it is not a real patient. Removing rows because they are inconvenient is how a model comes to look good on paper and fail in use, and the removal has to be declared and justified on grounds that would have applied before you saw the error figure, such as an impossible value or a duplicated record.
If the row is real, the honest options are to report both measures so the reader can see the effect, to fit on absolute error if the clinical cost genuinely is linear, or to model the extreme group separately.
How many decimal places should an error figure carry?
As many as the measurement supports and no more. A length of stay recorded in whole days does not support an error quoted to three decimal places, and doing so signals that the figure was copied from a console rather than thought about.
Round to the precision of the outcome, keep one extra digit only when you are comparing two models that differ in the last place, and in that case say that the difference is within the resampling spread if it is.
Assessment move
Take any fitted model, print its residual column, sort it, and write down the five largest. Then compute both error measures by hand on those ten numbers. Doing the arithmetic once fixes the relationship between the two measures far more firmly than reading it, and the sorted list is the evidence you need for the question about why you chose your measure.
Working through Measuring Error in a Regression Model in BMS5010? Sia is AskSia’s AI Artificial Intelligence tutor — ask any BMS5010 Measuring Error in a Regression Model question and get a clear, step-by-step explanation grounded in how BMS5010 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
