MATH7005 Chap.2 Cross-Entropy and Likelihood
Cross-Entropy and Likelihood
Loss reads probability assigned to truth
Binary cross-entropy selects the probability of the observed label and takes its negative logarithm. Correct confident predictions receive small losses, while confident mistakes dominate a batch.
The chapter derives positive-label and negative-label cases separately, then connects the sum of their losses to negative log-likelihood.
What to verify
A transparent calculation names the target label, selects the matching class probability, applies the logarithm and states whether losses were summed or averaged.
Stable implementations operate from logits or clip only at a carefully documented numerical boundary.
Read the records behind the mean
An average loss can rise because many predictions weakened slightly or because one confident error became extreme. Sort validation records by individual contribution and inspect the labels, input quality and subgroup context of the largest terms.
This turns a training curve into a diagnostic tool instead of treating the aggregate as an unexplained score.
Likelihood supplies the statistical bridge
Independent record probabilities multiply in a likelihood. Taking logarithms converts that product to a sum, and negating it produces the minimisation objective used in training.
State the independence and model assumptions rather than repeating that the two objectives are equivalent in every setting. A change in the probabilistic story changes the likelihood and therefore the corresponding loss.
Numerical care belongs in the method
Exactly zero probability on the observed class creates an undefined logarithm.
Combined logit-loss routines avoid first rounding sigmoid output and then logging it. When results are reported, name the reduction used, the number of records and whether class or sample weights changed the average. Those details determine whether two displayed losses can be compared.
What this chapter covers
- 01
Entropy and cross-entropy
- 02
Binary label cases
- 03
True-class probability
- 04
Batch sum and mean
- 05
Maximum likelihood connection
- 06
Logarithm stability
Worked example · free
Compute a mean binary loss
- 3Select both true-class probabilities.
- 3Take their negative logarithms.
- 2Average the losses.
Key terms
- Cross-Entropy
- A loss comparing a target distribution with predicted probabilities.
- True-Class Probability
- The probability the model assigned to the label that actually occurred.
- Likelihood
- The probability of the observed labels viewed as a function of the model parameters.
- Log-Likelihood
- The logarithm of likelihood, turning products across records into sums.
- Batch Mean
- The sum of record losses divided by the number of records in the batch.
Cross-Entropy and Likelihood FAQ
Why does a confident mistake cost so much?
Negative logarithm grows rapidly as the true-class probability approaches zero. The loss therefore distinguishes a hesitant error from a model that assigns almost all probability to the wrong outcome.
Are accuracy and cross-entropy interchangeable?
No. Accuracy keeps only the winning class, while cross-entropy retains probability confidence. Two models can classify the same records identically and still have very different losses because their probability assignments differ.
Why use negative log-likelihood?
Logarithms turn a product of record probabilities into a sum that is easier and more stable to optimise. The negative sign converts likelihood maximisation into the minimisation convention used by training algorithms.
Why inspect individual losses instead of only the mean?
One mislabeled or corrupted record can dominate a small validation average. Sorting records by contribution reveals whether deterioration is broad or concentrated and directs attention to labels, feature coverage and confident errors that can be checked directly.
What changes when class weights are added?
Weighted observations contribute more strongly to the training objective, so the result is not directly comparable with an unweighted mean. State the weighting rule and its operational reason, then evaluate ordinary product metrics beside the weighted loss.
How do I compare losses from different batch sizes?
Use the same reduction and data definition. A summed loss naturally grows with more records, while a mean divides by batch size. Also check whether sample weights, class weights or filtering changed which observations contribute.
What probability should enter the logarithm?
Use the probability assigned to the observed class, not automatically the positive-class probability. For a negative binary label that quantity is one minus p; for multiclass prediction it is the softmax entry indexed by the true label.
Assessment move
Build a tiny batch with both label values and write the positive-class probability beside the true-class probability for each row. Expand the binary formula by substituting the label before evaluating any logarithm. Sum the individual terms, then divide to obtain the mean and explain how the two reductions differ. Change one prediction from hesitant to confidently wrong and observe how its contribution dominates.
In code, compare a stable loss-from-logits function with the hand calculation and inspect the largest record losses rather than only the epoch average. For revision, derive the negative log-likelihood from a product of independent label probabilities and state which assumption allows the product. Keep natural logarithms and full precision until the final line.
Working through Cross-Entropy and Likelihood in MATH7005? Sia is AskSia’s AI Computer Science tutor — ask any MATH7005 Cross-Entropy and Likelihood question and get a clear, step-by-step explanation grounded in how MATH7005 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
