COMP4318 · Machine Learning and Data Mining
Linear & Logistic Regression; Overfitting & Regularization
Week 3 covers linear regression by least squares, logistic regression and the sigmoid for classification, and the overfitting problem with its regularization fixes (Ridge/L2 and Lasso/L1). It is the unit's first real dose of the bias–variance trade-off. In the exam it appears as a short least-squares or R² calculation, a conceptual overfitting question (why risk falls then rises, and two ways to avoid it), and definitions distinguishing Ridge from Lasso.
What this chapter covers
- 01Linear regression line ŷ = b0 + b1·x; least-squares slope b1 and intercept b0 from the sums
- 02Error sums SSE, SST, SSR and the identity SST = SSR + SSE; R² = SSR/SST in [0,1]
- 03Error metrics MAE, MSE, RMSE for train or test predictions
- 04Logistic regression for 2-class problems; the sigmoid/logistic curve P = e^(z)/(1 + e^(z)), z = b0 + b1·x
- 05Log-odds (logit) form ln(p/(1−p)) = b0 + b1·x; fit by maximum likelihood; threshold at 0.5
- 06Overfitting (low train, high test error) vs underfitting; the complexity–generalisation trade-off
- 07Ridge (L2): MSE + α·Σwᵢ² shrinks coefficients; Lasso (L1): MSE + α·Σ|wᵢ| can drive coefficients to exactly 0 (feature selection)
Fitting a least-squares line and predicting
- +1Compute the sums: n = 5, Σx = 15, Σy = 20, so x̄ = 3, ȳ = 4; Σxy = 2+6+15+16+30 = 69; Σx² = 1+4+9+16+25 = 55.
- +1Slope b1 = (Σxy − (Σx)(Σy)/n) / (Σx² − (Σx)²/n) = (69 − 15·20/5) / (55 − 15²/5) = (69 − 60)/(55 − 45) = 9/10 = 0.9.
- +1Intercept b0 = ȳ − b1·x̄ = 4 − 0.9·3 = 4 − 2.7 = 1.3.
- +1Fitted line: ŷ = 1.3 + 0.9·x.
- +1Predict at x = 6: ŷ = 1.3 + 0.9·6 = 1.3 + 5.4 = 6.7.
Key terms
- Linear regression
- Fitting a line ŷ = b0 + b1·x (a hyperplane with more features) to predict a numeric target; the coefficients are chosen by least squares.
- Least squares
- The fit that minimises the sum of squared errors SSE = Σ(yᵢ − ŷᵢ)²; gives b1 = (Σxy − (Σx)(Σy)/n)/(Σx² − (Σx)²/n) and b0 = ȳ − b1·x̄.
- R² (coefficient of determination)
- R² = SSR/SST ∈ [0,1]; the fraction of variance explained. R² = 1 is a perfect fit; on a poor test set it can be negative.
- Sigmoid / logistic function
- f(z) = 1/(1 + e^(−z)); the S-shaped curve mapping any real z to (0,1). Logistic regression models P(class 1) = e^(z)/(1 + e^(z)) with z = b0 + b1·x.
- Overfitting
- Low training error but high test error — the model memorised noise instead of generalising; underfitting is the too-simple opposite (poor on both).
- Ridge vs Lasso
- Regularized regression: Ridge adds an L2 penalty α·Σwᵢ² (shrinks coefficients); Lasso adds an L1 penalty α·Σ|wᵢ| and can set some coefficients to exactly 0, performing feature selection.
Linear & Logistic Regression; Overfitting & Regularization FAQ
What's the difference between Ridge and Lasso?
Both add a penalty on the coefficient sizes to the least-squares cost to fight overfitting. Ridge (L2) penalises Σwᵢ² and shrinks all coefficients smoothly toward zero without eliminating them. Lasso (L1) penalises Σ|wᵢ| and can drive some coefficients to exactly zero, so it also selects features. Larger α means stronger penalty and simpler models in both.
Why use logistic regression instead of linear regression for classification?
Linear regression predicts an unbounded number, which is awkward for a 0/1 class. Logistic regression passes the linear combination through the sigmoid to produce a probability in (0,1), then thresholds (usually at 0.5). It is fit by maximum likelihood, not least squares, and models the log-odds ln(p/(1−p)) = b0 + b1·x as linear in the features.
How is overfitting examined in COMP4318?
Usually as a short conceptual part: explain why the true risk (test error) first decreases then increases as model complexity grows, and give two ways to avoid it — regularization, more or more-representative training data, cross-validation, pruning/early stopping, or dimensionality reduction. Knowing the bias–variance trade-off in words earns the marks.
Can AI help me with the regression maths?
Yes. Sia can walk you through the least-squares sums line by line, explain why only the space of R² captures fit quality, and set fresh fitting or Ridge/Lasso questions to rehearse — checking your arithmetic and reasoning. It teaches the method and does not complete graded work for you.
Exam move
Practise the least-squares routine on small tables until the sums are automatic: Σx, Σy, Σxy, Σx², then b1 and b0, then a prediction and R². Keep the three error sums straight — SST = SSR + SSE and R² = SSR/SST — and be ready to explain the overfitting curve in words plus two remedies. For logistic regression, memorise the sigmoid and the logit form and know it is fit by maximum likelihood. Contrast Ridge (L2, shrinks) and Lasso (L1, zeroes/selects) in one line each. When the algebra slips, ask Sia to re-derive b1 with fresh numbers and check your working.
Working through Linear & Logistic Regression; Overfitting & Regularization in COMP4318? Sia is AskSia’s AI Machine Learning tutor — ask any COMP4318 Linear & Logistic Regression; Overfitting & Regularization question and get a clear, step-by-step explanation grounded in how COMP4318 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
