ECE5179 Chap.2 Logistic Regression and the Softmax Classifier
Logistic Regression and the Softmax Classifier
Week 2 replaces the straight-line fit with models that output probabilities. Logistic regression passes a linear score through the sigmoid and is trained with binary cross-entropy, whose gradient takes the simple error-times-input form. The softmax classifier extends the idea to many classes with one-hot labels and negative log-likelihood.
The chapter also shows why squared loss fails for labels, and why softmax depends only on differences between logits.
What this chapter covers
- 01
Why least squares fails on class labels
- 02
Hyperplanes, signed distance and the decision boundary
- 03
The sigmoid, odds and the logit
- 04
Binary cross-entropy and its gradient
- 05
One-hot labels and the softmax classifier
- 06
Shift invariance and the log-sum-exp trick
- 07
Temperature and label smoothing
Worked example · free
One logistic-regression update on a single sample
- 1Score z = 1.5 × 2 − 2 × 1 + 0.5 = 1.5, so the predicted probability is σ(1.5) ≈ 0.8176.
- 1Loss = −ln 0.8176 ≈ 0.2014.
- 1Error ŷ − y ≈ −0.1824, so the weight gradient is (−0.365, −0.182) and the bias gradient −0.182.
- 1Update: w ≈ (1.682, −1.909) and b ≈ 0.591 after subtracting 0.5 times each gradient.
- 1Check: the new score is about 2.05, so the probability of the true class rises to about 0.886.
Key terms
- Logit
- A logit is the raw score wᵀx + b, equal to the log-odds of the positive class in logistic regression.
- One-hot vector
- A one-hot vector encodes a class label as a vector that is zero everywhere except for a 1 at the true class.
- Log-sum-exp trick
- The log-sum-exp trick subtracts the largest logit before exponentiating so that softmax and its logarithm never overflow.
- Label smoothing
- Label smoothing softens each one-hot target slightly so that the loss gains a finite minimiser.
- Decision boundary
- A decision boundary is the set of inputs where a classifier's prediction changes from one class to another.
Logistic Regression and the Softmax Classifier FAQ
Why use cross-entropy instead of squared error for classification?
With a sigmoid output, squared error keeps a factor ŷ(1 − ŷ) in its gradient, so a confidently wrong prediction learns about a thousand times more slowly than under cross-entropy. Cross-entropy is also convex in the weights, so gradient descent has no poor local minima to fall into for this model.
Is softmax really unchanged when I add a constant to all logits?
Yes. Each exponential gains the same factor, which cancels between numerator and denominator, so only differences between logits matter. Implementations use this to subtract the largest logit before exponentiating, which prevents overflow without changing the probabilities.
How does the softmax classifier relate to logistic regression?
With two classes, softmax depends only on the difference between the two logits, and its first output equals the sigmoid of that difference. Logistic regression is therefore the two-class special case, with one effective weight vector equal to the difference of the two class weight vectors.
Why does the sigmoid not bend the decision boundary?
The sigmoid is monotone, so σ(z) ≥ 0.5 exactly when z ≥ 0. The boundary is therefore the set where wᵀx + b = 0, which is still a flat hyperplane. The sigmoid only turns the distance from that plane into a probability; curved boundaries need extra features or hidden layers.
What does label smoothing change in training?
It replaces each one-hot target with a mixture that gives the true class 1 − ε plus a share ε/K to every class. The loss then has a finite minimiser, so the weights stop growing without bound on separable data and the model's confidences stay meaningful.
Exam move
Derive the cross-entropy gradient once from σ′ = σ(1 − σ) until the cancellation feels natural, then practise single-sample updates with new numbers, writing score, probability, loss, gradient and update on separate lines. For softmax, compute probabilities, loss and the logit gradient p − y for a three-class example, and verify shift invariance by adding a constant to every logit.
Finish with a geometry question: draw the boundary of a two-feature classifier, classify two points and compute one signed distance. Keep a short list of identities to recall under time pressure: σ(−z) = 1 − σ(z), σ′ = σ(1 − σ), the logit as log-odds, softmax shift invariance and the gradient p − y.
Most derivations in this chapter are two or three of these chained together, and recognising which ones apply is faster than deriving from scratch.
Working through Logistic Regression and the Softmax Classifier in ECE5179? Sia is AskSia’s AI Computer Science tutor — ask any ECE5179 Logistic Regression and the Softmax Classifier question and get a clear, step-by-step explanation grounded in how ECE5179 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
