BME5150 Chap.5 Ridge, Lasso and Elastic Net
Ridge, Lasso and Elastic Net
Regularisation changes the fitting objective
Least squares rewards close fit to development outcomes. Regularisation adds a cost for coefficient size. Lambda controls how strongly that cost competes with fit. At zero, the solution matches the unpenalised fit.
As lambda increases, coefficients shrink, training error usually rises and held-out stability may improve.
Ridge uses an L2 penalty
Ridge adds the sum of squared coefficients. Squaring makes a large coefficient contribute strongly to the penalty. Correlated predictors often share influence and coefficients typically approach zero without becoming exactly zero.
Ridge can stabilise predictions when many variables contain overlapping information.
Lasso uses an L1 penalty
Lasso adds the sum of absolute coefficient values. Its geometry permits coefficients to reach exactly zero, creating a sparse predictive model. Which variable survives among correlated alternatives can change with the sample.
A zero coefficient therefore means the fitted specification did not retain that variable, not that the underlying phenomenon is absent.
Elastic net combines both behaviours
Elastic net mixes L1 and L2 penalties. It can retain grouped shrinkage while still producing sparsity. The mixing parameter and lambda both require tuning.
That flexibility is useful for correlated radiomic features and increases the need for nested development resampling.
Scaling and validation complete the method
Penalties act on coefficient magnitudes, so predictors need comparable treatment. Fit scaling within training data.
Choose lambda from held-out development folds rather than training loss, refit the selected specification on all development cases and evaluate the locked pipeline once.
What this chapter covers
- 01
Penalised regression objectives
- 02
Lambda and coefficient shrinkage
- 03
Ridge L2 behaviour
- 04
Lasso L1 sparsity
- 05
Elastic net mixing
- 06
Scaling, tuning and selection stability
Worked example · free
Compare L1 and L2 penalty shapes
- 3Compute squared and absolute coefficient sums.
- 2Multiply each by lambda.
- 3Interpret the penalty shapes without comparing tuned lambda scales directly.
Key terms
- Regularisation
- Regularisation adds a complexity penalty to a fitting objective to improve generalisation.
- Lambda
- Lambda controls the strength of the penalty relative to model fit.
- L2 Penalty
- An L2 penalty sums squared coefficients and produces continuous shrinkage.
- L1 Penalty
- An L1 penalty sums absolute coefficients and can produce exact zeros.
- Elastic Net
- Elastic net combines L1 and L2 penalties through a tuned mixture.
Ridge, Lasso and Elastic Net FAQ
Why should predictors be scaled before ridge or lasso?
Penalty size depends on coefficient magnitude, and coefficient magnitude depends on predictor units. Without deliberate scaling, the model penalises variables differently for unit reasons. Fit scaling within each development fold and apply the stored parameters to held-out data.
Does lasso prove that a removed feature is unimportant?
No. It shows that one fitted penalised specification did not retain the feature under a chosen scale, lambda and set of correlated alternatives. Check how often the feature survives resampling and use independent evidence for biological importance.
How should lambda be chosen?
Choose lambda by the metric that matches the task within development cross-validation. Training loss usually favours too little penalty. After selection, refit the complete specification on development data and apply it once to the untouched evaluation set.
Assessment move
Draw four coefficient paths as lambda moves right: two ridge paths approaching zero and two lasso paths reaching zero. Explain why the picture changes only after predictors are placed on a comparable scale. Recompute L1 and L2 penalties for small coefficient vectors until the effect of squaring is immediate. Then practise the valid workflow from split through nested scaling and tuning to final evaluation.
For interpretation, take one variable selected by lasso and write a safe predictive statement, a stability question and a biological claim that would require separate evidence. Finish by explaining when elastic net might be preferred for groups of correlated radiomic features. Compare the selected feature set across imagined folds and decide what selection frequency would make you cautious.
The point is to distinguish sparse prediction from a stable scientific signature.
Working through Ridge, Lasso and Elastic Net in BME5150? Sia is AskSia’s AI Computer Science tutor — ask any BME5150 Ridge, Lasso and Elastic Net question and get a clear, step-by-step explanation grounded in how BME5150 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
