MATH7005 Chap.3 Softmax and Multiclass Prediction
Softmax and Multiclass Prediction
One distribution from competing scores
Softmax exponentiates a vector of class logits and divides by one shared sum, producing positive probabilities that add to one.
The chapter subtracts the largest logit before exponentiation, tracks the class axis through matrix shapes and reads categorical loss at the observed-class index.
What to verify
Every probability row should sum to one, adding a common constant to all logits should change nothing, and normalisation must occur across classes rather than across unrelated observations.
The full probability vector remains valuable even when only the winning class drives the product.
Class indexing is part of correctness
A probability vector has meaning only through its class mapping. Preserve the training label order, export it with the model and test a record whose category is easy to recognise.
An off-by-one or reordered label can attach a correct calculation to the wrong product action, so the mapping deserves the same validation as the arithmetic.
Confidence needs context
The top probability and the margin to the runner-up reveal more than argmax alone, but neither guarantees calibration.
Compare predicted confidence with observed frequency on held-out data and inspect whether particular classes are systematically overconfident. If uncertain cases are routed to review, report the workload produced by that rule and the important errors it still misses.
Shapes protect the normalisation axis
Write batch size and class count beside the logit matrix before calling softmax.
A rowwise sum check detects accidental normalisation down the batch, while shift invariance detects unstable or incomplete code. For a multi-label target, replace the entire output assumption with independent sigmoids rather than trying to repair softmax after training.
What this chapter covers
- 01
Class logits
- 02
Softmax normalisation
- 03
Maximum subtraction
- 04
Categorical cross-entropy
- 05
Class-axis dimensions
- 06
Top-two decision margin
Worked example · free
Normalise three logits stably
- 2Subtract the largest logit.
- 3Exponentiate and sum.
- 3Divide and identify the maximum.
Key terms
- Logit Vector
- The unrestricted class scores produced before multiclass normalisation.
- Softmax
- A rowwise transformation turning logits into one categorical probability distribution.
- Class Axis
- The tensor dimension across which mutually exclusive class probabilities are normalised.
- Categorical Loss
- The negative logarithm of the probability assigned to the observed class.
- Decision Margin
- The gap between the leading and next class score or probability.
Softmax and Multiclass Prediction FAQ
Why subtract the largest logit?
The operation prevents a large positive exponential from overflowing and leaves every softmax probability unchanged because the same multiplicative factor cancels from numerator and denominator.
When should I use independent sigmoids instead?
Use independent sigmoids when several labels may be true simultaneously. Softmax is appropriate when the classes are mutually exclusive and should compete for one unit of probability mass.
Can I normalise down the batch dimension?
Not for ordinary multiclass prediction. That would make different records compete with each other. Each record needs its own distribution across classes, so softmax acts along the class axis within every row.
How do I keep class labels aligned with probabilities?
Export the training label order with the model and test a record whose class is easy to recognise. A probability vector is only meaningful through that mapping, and an off-by-one index can turn correct arithmetic into a wrong product action.
What does a small top-two margin tell me?
It shows that the leading classes are locally competitive, but it does not prove that the probabilities are calibrated. Evaluate any review rule on held-out data and report both the routed workload and the important errors left outside it.
Why inspect a class-level confusion matrix?
An aggregate score can be dominated by common classes. The matrix reveals which labels are substituted for each other, supports per-class recall calculations and helps connect uncertain probability vectors with the particular product errors they create.
Does adding a constant change a softmax prediction?
Adding the same constant to every logit changes neither the probabilities nor the winning class because the common exponential factor cancels. This invariance justifies subtracting the row maximum as a numerical-stability step.
Assessment move
Create one three-class example and calculate stable softmax by subtracting the maximum, exponentiating and dividing by the common sum. Check that the row totals one and repeat after adding a large constant to every logit. Write the observed class index beside the vector before taking categorical loss. In code, preserve the class-name mapping and assert the intended normalisation axis.
Compare two records with the same winning class but different top-two margins, then ask whether either confidence level is calibrated on held-out data. Finally, contrast a mutually exclusive target with a multi-label target and explain why the first uses softmax while the second uses independent sigmoids. These checks should be runnable in the product demonstration.
Working through Softmax and Multiclass Prediction in MATH7005? Sia is AskSia’s AI Computer Science tutor — ask any MATH7005 Softmax and Multiclass Prediction question and get a clear, step-by-step explanation grounded in how MATH7005 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
