DATA2002 Chap.13 Linear regression and model interpretation
Linear regression and model interpretation
Module 4 changes the goal. Until now the explanatory variable has been a label naming a group; from here it can be a measurement, and the object of interest is a relationship rather than a set of differences. It also changes what a good answer looks like, because a model that describes the sample beautifully may predict a new observation badly. The model itself does not claim the points lie on a line.
It claims the mean response lies on a line and that individual observations scatter about it symmetrically, independently, and with a spread that does not change along the range. Those four claims are the four assumptions, each with its own diagnostic plot, and the unit is blunt about the order of work: a p-value means nothing if you have not looked at your data.
Least squares chooses the line minimising the total squared vertical distance, and two consequences are worth carrying. The fitted line always passes through the point of means, and the divisor in the variance estimate is the sample size minus two because two parameters were estimated before the residuals could be formed. The same fitted line then plays two roles.
The slope answers a question about the population, how much the mean response changes per unit of the explanatory variable, and it is tested against zero with a t statistic.
The fitted values answer a question about cases, and here the chapter's most examined distinction appears: a confidence interval for the mean response at a given value is narrower than a prediction interval for a single new observation, because the second must carry the scatter of individual points as well as the uncertainty in the line. That extra term does not shrink as the sample grows.
Finally, r-squared measures fit and cannot select a model, because adding any predictor can only reduce the residual sum of squares.
What this chapter covers
- 01
The model as a claim about the mean response rather than about individual points
- 02
Least squares, and why the distances measured are vertical
- 03
The fitted line through the point of means, and the divisor of the variance estimate
- 04
Residuals as the raw material for every check and every standard error
- 05
Whether the intercept is interpretable, and the extrapolation test for deciding
- 06
The four assumptions and the plot that checks each
- 07
The one plot that does two jobs, and what a describable pattern in it means
- 08
Transformations, and the three different readings of a transformed coefficient
- 09
Inference on the slope, and why the degrees of freedom are the sample size minus two
- 10
The decomposition again, r-squared, and the adjusted version
- 11
Why r-squared can never fall when a predictor is added
- 12
Confidence against prediction intervals, and the extra term that never shrinks
- 13
Four earlier tests rewritten as one linear model, and what plays the part of Y in each
- 14
Where the equivalence is exact and where it is only close
Read two fit measures and say what they tell you
- +1.5With thirty two observations, two predictors and an intercept, three parameters are estimated, so the residual degrees of freedom are thirty two minus two minus one, which is twenty nine.
- +1.5The rise in r-squared is uninformative. Adding any predictor at all can only reduce the residual sum of squares, because the previous model remains available as a special case, so r-squared can never fall.
- +1The fall in the adjusted value is informative: it applies a penalty for the extra parameter, and that penalty outweighing the gain in fit is evidence the third predictor is not earning its place.
- +1Next, compare the two models on a criterion that penalises complexity or, better, on out of sample performance, because both r-squared measures are computed on the data the model was fitted to and are therefore optimistic.
Key terms
- Least squares
- The criterion that chooses the line minimising the sum of squared vertical distances from the observations to it. Measuring perpendicular distances instead would answer a different question.
- Fitted value
- The value the model predicts at a given value of the explanatory variable. The collection of fitted values is the fitted line evaluated at the observed points.
- Residual
- An observation minus its fitted value. Residuals are not measurement errors to be apologised for; they are what every diagnostic and every standard error is computed from.
- Residual standard error
- The estimated typical size of a residual, computed with a divisor of the sample size minus the number of estimated parameters. It sets the scale of everything the model cannot explain.
- Extrapolation
- Using a fitted model outside the range of explanatory values observed. The intercept is an extrapolation whenever zero is not a plausible value, in which case it is a number the line needs rather than a finding.
- Constant variance
- The assumption that the spread of the errors does not depend on the fitted value. A widening funnel in the residual plot is the classic failure and often responds to a transformation.
- R-squared
- The proportion of the total variation in the response accounted for by the model. It measures fit and cannot select a model, because it can never fall when a predictor is added.
- Adjusted R-squared
- A version that divides each sum of squares by its own degrees of freedom, so that it can fall when a useless predictor is added. It is still computed in sample and is still optimistic.
- Confidence interval for the mean
- An interval for the average response at a stated value of the explanatory variable, whose width comes only from uncertainty in the fitted line.
- Prediction interval
- An interval for a single new observation at a stated value, which additionally carries the scatter of individual points about the line and is therefore always wider.
- Intercept only model
- A linear model with no explanatory variable at all, so the one coefficient estimated is the mean of the response. Fitting one and testing that coefficient against zero is arithmetically the one sample t test, and its interval is the t interval.
- Signed rank transform
- Replacing each value by the sign of its distance from a reference point times the rank of the size of that distance. A model fitted to the transformed values tracks the Wilcoxon signed rank test closely, and more closely as the sample grows.
Linear regression and model interpretation FAQ
Which interval does a question want?
The wording tells you. Estimate the average response for cases with this value asks for the confidence interval. Predict the response for a new case with this value asks for the prediction interval. The word average or mean against the word predict or a new individual case is the discriminator, and it is often the only difference between two otherwise identical parts of a question.
The prediction interval is always the wider of the two.
Why does the prediction interval not shrink to nothing with more data?
Because it carries two sources of uncertainty and only one of them depends on the sample size. The uncertainty in the fitted line does shrink as observations accumulate, so the confidence band collapses towards the line. The scatter of an individual observation about the line is a property of the population and does not shrink at all, so the prediction band has a floor set by the residual spread.
Any answer claiming both intervals shrink to nothing has missed the extra term.
Is the intercept worth interpreting?
Only when a value of zero for the explanatory variable is meaningful and lies inside or close to the observed range. If the explanatory variable runs from four to eighteen, the intercept is an extrapolation to a value never seen, and it should be treated as a number the line requires rather than as a finding.
Extending the axes back to zero and looking at where the line lands is the quick way to see whether the question is even sensible.
What does holding the other variables constant actually add?
It is the difference between a coefficient in a multiple regression and the plain association between that predictor and the response, and the two can even have opposite signs. The coefficient describes the change in the mean response per unit of that predictor with the rest of the model in place, which is a conditional statement.
Omitting the phrase from an interpretation loses the mark, and including it while describing an unconditional association is worse.
What do I do about a curved residual plot?
Take it as evidence that the straight line model has missed the shape, and consider modelling a transformed response. A curved pattern together with a widening spread very often responds to using the logarithm of the response.
That is a genuine fix rather than a cosmetic one, and it has a price: the model is now about the transformed scale, so the coefficient must be interpreted accordingly and the conclusion must say which scale it is on.
Why would anyone fit a model when a named test already answers the question?
Because it is one answer learned once instead of four learned separately. A one sample t test, a pooled two sample t test and their two rank based counterparts are the same linear model with different things put in the place of the response and of the explanatory variable. Learning the model gives you all four, and it tells you what to do when a scenario is a mixture that none of the four named procedures covers.
It also explains why a printed coefficient table can answer a question that never says regression anywhere: the row you are asked to read may be a two sample comparison wearing a different name. Two of the four correspondences are exact and the two rank based ones are approximate, so a p-value taken from the model should not be reported as if the rank test had produced it.
Exam move
Fix the order of work first, because it is the thing the unit states most bluntly: fit, plot the residuals, decide whether the model is admissible, and only then read the inference. Build the habit by never letting yourself look at a p-value in a practice question until you have written what you would check.
Second, learn the four assumptions as four pairs, each an assumption with its plot and the shape that indicates failure, since a question naming a diagnostic is asking which assumption it speaks to. Third, drill the two intervals until the discriminating word triggers the right one automatically, and be able to say in one sentence why the prediction interval is wider, because that sentence is the mark.
Fourth, memorise the degrees of freedom as the sample size minus the number of predictors minus one, and check it against any printed output you meet, since it is a fast way to confirm how many predictors a model contains. Finally, practise the interpretation sentence out loud with the two required qualifiers in it, mean response and holding the other variables constant, until leaving one out feels wrong.
Working through Linear regression and model interpretation in DATA2002? Sia is AskSia’s AI Statistics tutor — ask any DATA2002 Linear regression and model interpretation question and get a clear, step-by-step explanation grounded in how DATA2002 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
