COMP5318 Chap.3 Linear & Logistic Regression, Regularization
Linear & Logistic Regression, Regularization
This Week 3 topic of COMP5318 Machine Learning and Data Mining at the University of Sydney builds your first explicit parametric models: linear regression fitted by least squares and scored with R², and logistic regression, which sends the same linear score through a sigmoid to predict a class probability.
It closes with regularization — the Ridge (L2) and Lasso (L1) penalties that shrink weights to fight overfitting — all of which show up as short numeric problems in the closed-book final.
What this chapter covers
- 01
Fit a straight line ŷ = b₀ + b₁x by least squares using the closed-form slope and intercept formulas
- 02
Understand why least squares minimises SSE (squared residuals), not the raw sum of residuals
- 03
Split the spread of y with the identity SST = SSR + SSE and read each sum
- 04
Compute R² = SSR/SST = 1 − SSE/SST and interpret it as the fraction of variation explained
- 05
Recover the correlation r = ±√R², taking the sign from the slope direction
- 06
See how logistic regression turns the linear score into a probability with the sigmoid and the log-odds link
- 07
Apply the p ≥ 0.5 decision rule and find the decision boundary at z = 0
- 08
Know logistic regression is fitted by maximum likelihood, not least squares
- 09
Contrast Ridge (L2) shrinkage with Lasso (L1) feature selection and the role of the penalty strength α
Fit a least-squares line, predict, and compute R²
- +1Sums (n = 4): Σx = 10, Σy = 10, so x̄ = 2.5, ȳ = 2.5. Σxy = 1 + 6 + 6 + 16 = 29. Σx² = 1 + 4 + 9 + 16 = 30.
- +1Slope: b₁ = [Σxy − (Σx)(Σy)/n] / [Σx² − (Σx)²/n] = [29 − (10)(10)/4] / [30 − 100/4] = (29 − 25)/(30 − 25) = 4/5 = 0.8.
- +1Intercept: b₀ = ȳ − b₁x̄ = 2.5 − 0.8·2.5 = 2.5 − 2.0 = 0.5. Fitted line: ŷ = 0.5 + 0.8x.
- +1Predict x = 5: ŷ = 0.5 + 0.8·5 = 0.5 + 4.0 = 4.5.
- +1SSE: fitted ŷ = 1.3, 2.1, 2.9, 3.7; residuals −0.3, 0.9, −0.9, 0.3; squares 0.09, 0.81, 0.81, 0.09 → SSE = 1.80.
- +1SST and R²: deviations of y from ȳ = 2.5 are −1.5, 0.5, −0.5, 1.5; squares 2.25, 0.25, 0.25, 2.25 → SST = 5.00. R² = 1 − 1.80/5.00 = 1 − 0.36 = 0.64.
Key terms
- Least squares
- The criterion that chooses the line's slope and intercept to minimise the sum of squared residuals (SSE); it gives the closed-form b₁ and b₀ formulas.
- Residual
- The leftover error for one point, eᵢ = yᵢ − ŷᵢ — the vertical gap between the observed value and the fitted line.
- SSE / SST / SSR
- SSE = Σ(y−ŷ)² (unexplained), SST = Σ(y−ȳ)² (total spread), SSR = Σ(ŷ−ȳ)² (explained). They satisfy SST = SSR + SSE.
- R² (coefficient of determination)
- R² = SSR/SST = 1 − SSE/SST, the fraction of the spread in y the model explains; on the fitting data it runs 0 to 1, but can go negative on a held-out set.
- Sigmoid (logistic function)
- σ(z) = 1/(1+e⁻ᶻ), which maps any real score z into a probability in (0, 1); it equals 0.5 exactly at z = 0.
- Log-odds (logit)
- ln(p/(1−p)) = b₀ + b₁x — the log of the odds, which logistic regression models as a linear function of x even though p itself is not linear.
- Ridge (L2)
- Regularization that adds α·Σwⱼ² to the error; it shrinks all weights smoothly toward 0 but never sets any exactly to 0.
- Lasso (L1)
- Regularization that adds α·Σ|wⱼ| to the error; it can drive some weights exactly to 0, performing automatic feature selection. α (or λ) sets the penalty strength.
Linear & Logistic Regression, Regularization FAQ
Is linear regression fitted the same way as logistic regression?
No — and the exam tests the difference. Linear regression fits a numeric-target line by least squares (minimising SSE), which has a closed form. Logistic regression is a two-class classifier: it runs the linear score through a sigmoid and chooses the weights that make the observed 0/1 labels most probable — maximum likelihood, solved iteratively. Writing 'logistic regression minimises SSE' is a common false statement.
What is the difference between Ridge and Lasso, and when does R² go negative?
Ridge (L2) adds α·Σwⱼ² and shrinks weights smoothly toward 0 without ever zeroing them; Lasso (L1) adds α·Σ|wⱼ| and can set some weights exactly to 0, so it selects features. A larger α means a heavier penalty and a simpler model. R² sits in [0, 1] only on the data the model was fitted to; scored on a different test set, a poorly-fitting model can predict worse than the mean and give a negative R².
Can AI help me with linear and logistic regression in COMP5318?
Yes, for understanding. Sia can explain each step — how the least-squares formulas are derived, why the sigmoid outputs a probability, or how Ridge and Lasso differ — and walk you through a practice problem with your own numbers so you learn the method.
Use it to check your reasoning and rehearse the derivations, not to obtain answers to submitted assignments or the closed-book exam; the unit requires you to acknowledge any AI tools used in assessable work, so keep AI to learning and revision.
Exam move
Treat this topic as two formulas-plus-a-penalty. First drill the least-squares pipeline until it is automatic: lay out columns for x, y, xy and x², compute the slope and intercept, then the residuals, SSE, SST and R² — practise on small four- or five-point datasets so a full fit-and-score takes only a few minutes.
Second, lock the logistic side: the sigmoid formula, the log-odds link, the p ≥ 0.5 rule (boundary at z = 0), and the fact that it is fitted by maximum likelihood, not least squares. Third, memorise the one contrast the exam always frames — Ridge (L2) shrinks, Lasso (L1) selects — and what the strength α trades.
The final is 2 hours, closed book, with a non-programmable calculator only; regression appears as a short numeric problem, so budget roughly one minute per mark (a 4-mark R² part is about 4 minutes) and always show the formula before the number. Keep AI tutoring to rehearsing the method, and confirm the exact exam date on the Canvas exam timetable.
Working through Linear & Logistic Regression, Regularization in COMP5318? Sia is AskSia’s AI Computer Science tutor — ask any COMP5318 Linear & Logistic Regression, Regularization question and get a clear, step-by-step explanation grounded in how COMP5318 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
