ECE5179 Chap.1 Learning Types and Linear Regression
Learning Types and Linear Regression
Week 1 sets out the kinds of learning problem, from supervised to self-supervised, traces the artificial neuron back to the McCulloch-Pitts model, and then develops linear regression as the first trainable model.
The chapter derives the closed-form least-squares line from the covariance and variance of the data, extends it to many features through the normal equations, and explains how the bias, the residuals and the shape of the data matrix decide which solution applies.
What this chapter covers
- 01
Supervised, unsupervised, semi-supervised and self-supervised learning
- 02
Regression versus classification targets
- 03
The McCulloch-Pitts neuron and the modern artificial neuron
- 04
Residuals and the mean squared error
- 05
Closed-form slope and intercept from covariance and variance
- 06
Normal equations, the pseudoinverse and bias augmentation
- 07
Overdetermined and underdetermined systems
Worked example · free
Fit a least-squares line to five points
- 1Means: x̄ = 3 and ȳ = 4.0.
- 1Covariance term: the products of deviations add to 9.7, so σ̂xy = 9.7/5 = 1.94.
- 1Variance term: squared x deviations add to 10, so σ̂xx = 2.
- 1Slope w = 1.94/2 = 0.97 and intercept b = 4.0 − 3 × 0.97 = 1.09.
- 1Predictions 2.06, 3.03, 4.00, 4.97, 5.94 give residuals −0.04, +0.13, −0.20, +0.17, −0.06.
- 1Squared residuals sum to 0.091, so the mean squared error is 0.0182.
Key terms
- Supervised learning
- Supervised learning fits a model to inputs that each come with a known target label.
- Residual
- A residual is the difference between a model's prediction and the true target for one sample.
- Normal equations
- The normal equations XᵀXw = Xᵀy are the conditions that the least-squares weights must satisfy.
- Bias augmentation
- Bias augmentation prepends a constant 1 to each input so the intercept becomes an ordinary weight.
Learning Types and Linear Regression FAQ
Why does the least-squares line pass through the mean point?
Setting the derivative of the mean squared error with respect to the intercept to zero forces the residuals to average zero. That condition gives b = ȳ − w x̄, and substituting x̄ into the line returns ȳ exactly. Checking this property is a quick way to catch an arithmetic slip in a hand-fitted line.
When can I not use the normal-equation formula?
The formula needs XᵀX to be invertible, which fails when features outnumber samples or when some features are exact combinations of others. With more features than samples the lecture notes switch to the minimum-norm solution, which fits the data exactly while keeping the weights as small as possible.
Does a large regression weight mean a feature causes the target?
No. A weight measures how the prediction changes with a feature while the others stay fixed, and that only compares fairly across features after standardising them. Even then it describes association in the fitted model, not causation, which the lecture notes stress explicitly.
How is self-supervised learning different from unsupervised learning?
Both start without human labels, but self-supervised learning creates a prediction target from the data itself, for example hiding part of an input and predicting it. That turns the problem into a supervised one with automatically generated labels, which is the idea later used to pre-train large language models and encoders such as BERT.
What did the McCulloch-Pitts neuron contribute?
Proposed in 1943, it modelled a neuron as a unit with binary inputs that fires when enough excitatory inputs are active and no inhibitory input vetoes it. Networks of such units can realise AND, OR and NOT and therefore any logical function, the starting point for the weighted-sum-and-activation neurons used today.
Exam move
Fit at least one small data set completely by hand: means, deviations, covariance, variance, slope, intercept, residuals and error, keeping every intermediate sum visible. Then redo it from summary statistics alone, which is how studio questions often present the data.
For the matrix form, practise stating the shape of every object, X, Y, XᵀX and W, for a few made-up sizes before touching any formula, and decide each time whether the system is over- or underdetermined. Finish by explaining in one sentence what the slope means in the units of the problem.
A useful self-test is to change one data point and predict, before recomputing, whether the slope rises or falls; the sign of that point's deviation product tells you. When the notes switch to matrices, translate each scalar step into its matrix counterpart: means become column averages, the covariance ratio becomes XᵀX and Xᵀy, and the intercept becomes the weight on the augmented column of ones.
Working through Learning Types and Linear Regression in ECE5179? Sia is AskSia’s AI Computer Science tutor — ask any ECE5179 Learning Types and Linear Regression question and get a clear, step-by-step explanation grounded in how ECE5179 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
