QBUS2820 Chap.7 Ridge Regression
Ridge Regression
Ridge regression stabilises a linear model by adding a squared-coefficient penalty to residual sum of squares. It is especially useful when predictors are highly correlated and ordinary least-squares slopes change sharply across samples. Lambda controls the bias-variance compromise: zero reproduces ordinary least squares, while larger values shrink slopes toward zero and reduce effective complexity.
Correct use requires training-only standardisation, an unpenalised intercept in the usual setup, and tuning by cross-validation or a valid penalised information criterion. The geometry of ridge can be understood through correlated directions in predictor space. Ordinary least squares may use large opposing slopes to fit along a direction with little information; the squared penalty discourages that unstable combination.
Effective degrees of freedom fall smoothly as lambda increases, providing a way to connect shrinkage with model complexity. Coefficient paths should be inspected for sign stability and abrupt movement near the selected penalty. Standardisation is not a cosmetic preliminary because the penalty directly compares coefficient magnitudes. The fitted centring and scale parameters must travel with the deployed model.
Ridge can be strong when many predictors carry modest signal, but it does not supply a short explanatory list. Prediction robustness should also be tested under plausible shifts in predictor correlation, since regularisation stabilises the training problem rather than guaranteeing invariance. Where several nearby lambda values have similar error, a stable region may be more defensible than the exact numerical minimum.
What this chapter covers
- 01
Multicollinearity and instability
- 02
Squared coefficient penalties
- 03
Lambda and shrinkage paths
- 04
Bias and variance trade-off
- 05
Predictor standardisation
- 06
Cross-validated lambda choice
- 07
Effective degrees of freedom
- 08
Prediction compared with explanation
Worked example · free
Select a ridge penalty
- +1The marks shown for this tuning exercise are not an official University assessment allocation. Compare the held-out losses on the same folds and preprocessing pipeline.
- +1The smallest error is 10.6 at λ = 0.2.
- +1Select λ = 0.2 and refit using the training data under that tuning rule.
- +1Interpret smaller slopes as shrinkage toward zero, not automatic feature deletion.
- +1Evaluate once on untouched test cases before reporting final performance.
Key terms
- Multicollinearity
- Strong predictor dependence that can make individual least-squares slopes unstable.
- Ridge Penalty
- Lambda times the sum of squared slopes added to residual fit.
- Shrinkage
- Movement of fitted coefficients toward zero to reduce prediction variance.
- Tuning Parameter
- A value such as lambda selected through validation rather than ordinary fitting.
- Coefficient Path
- A plot showing fitted slopes across a range of penalty strengths.
- Effective Complexity
- The flexible capacity remaining after shrinkage, often expressed through effective degrees of freedom.
- Standardised Coefficient
- A slope fitted after scaling its predictor, useful for understanding shrinkage on a common input scale.
Ridge Regression FAQ
Why does ridge help with correlated predictors?
The squared penalty discourages unstable large coefficient combinations. Ridge accepts some bias but can sharply reduce variance, producing more reliable predictions when ordinary least-squares slopes compete across correlated inputs.
Does ridge perform variable selection?
Ordinary ridge regression shrinks slopes smoothly toward zero but generally does not make them exactly zero. It stabilises dense predictive models rather than returning a sparse subset of selected variables.
Why must predictors be standardised?
The penalty acts on coefficient size, which depends on predictor units. Standardisation prevents measurement scale from deciding which slopes receive stronger effective shrinkage during model fitting.
What does effective degrees of freedom mean here?
It describes the complexity remaining after ridge shrinkage. As lambda increases, directions associated with the predictor matrix contribute less, so effective complexity falls smoothly rather than by deleting whole variables.
Can ridge coefficients be compared directly?
Only after considering predictor scale. Standardised slopes share a common input scale, while original-unit slopes answer operational questions. Report which version is being interpreted and preserve the transformation for deployment.
Exam move
Begin with a two-predictor collinearity example and explain why several slope combinations can fit similarly. Draw coefficient paths moving toward zero as lambda increases. Write the ridge objective from memory and label the fit term, penalty and tuning parameter. Practise choosing lambda from validation error rather than residual variance, then explain why the intercept is not usually penalised.
Reproduce the source-backed AIC logic for ridge effective degrees of freedom with a small table, but keep the main interpretation focused on out-of-sample loss. Compare standardised and original-unit coefficients and document how a deployment system must reuse the training means and scales.
End with a paragraph separating prediction stability from causal interpretation: good ridge prediction under correlated inputs does not identify one predictor's isolated real-world effect.
Working through Ridge Regression in QBUS2820? Sia is AskSia’s AI Statistics tutor — ask any QBUS2820 Ridge Regression question and get a clear, step-by-step explanation grounded in how QBUS2820 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
