QBUS2820 Chap.10 Classification Loss and ROC Analysis
Classification Loss and ROC Analysis
Classification evaluation starts with the four confusion-matrix cells and the identity of the positive class. Sensitivity, specificity, false-positive rate and false-negative rate describe different error directions. Misclassification rate treats all mistakes equally, while a loss matrix prices their consequences.
ROC curves examine sensitivity and false-positive rate across thresholds, and AUC summarises ranking rather than calibration or business value. A defensible operating threshold combines held-out performance with prevalence, costs, benefits and capacity. Every rate begins from a population of cases in one actual class. Sensitivity divides by actual positives; specificity divides by actual negatives.
Precision answers a different question by dividing true positives by all predicted positives and therefore depends strongly on prevalence. A loss matrix goes beyond rates by valuing each actual-action outcome, allowing expected cost or benefit to be computed for a probability and threshold.
ROC curves remain unchanged by some prevalence shifts because they condition on actual classes, but the economic value of an operating point does not. Capacity constraints can be included by restricting the set of feasible thresholds. AUC summarises pairwise ranking and can conceal poor performance in the narrow false-positive region an operation actually uses.
Threshold selection must occur on validation evidence, followed by one final assessment. After deployment, monitor confusion counts, prevalence, calibration and realised consequences because a stable AUC does not guarantee a stable business decision.
What this chapter covers
- 01
Confusion matrix cells
- 02
Sensitivity and specificity
- 03
False-positive and false-negative rates
- 04
Misclassification error
- 05
Unequal error costs
- 06
Loss matrices and expected value
- 07
ROC curves and AUC
- 08
Threshold selection under constraints
Worked example · free
Evaluate a fraud threshold
- +1The marks displayed for this metric exercise are a study device, not a University published mark allocation. TP = 32 and FN = 40 − 32 = 8.
- +1FP = 48 and TN = 960 − 48 = 912.
- +1Sensitivity = 32/40 = 0.80.
- +1Specificity = 912/960 = 0.95.
- +1Accuracy = (32 + 912)/1000 = 0.944, but the rare positive class makes class-specific metrics essential.
Key terms
- Confusion Matrix
- A table crossing predicted and actual classes into four outcome counts.
- Sensitivity
- The fraction of actual positives correctly detected as positive.
- Specificity
- The fraction of actual negatives correctly classified as negative.
- Loss Matrix
- A table assigning costs or benefits to actual-predicted outcome combinations.
- ROC Curve
- Sensitivity plotted against false-positive rate as the threshold changes.
- AUC
- A threshold-free summary of class ranking performance under an ROC curve.
- Operating Point
- One threshold-specific location on an ROC curve with a particular sensitivity and false-positive rate.
Classification Loss and ROC Analysis FAQ
Why can accuracy be misleading?
When one class is rare, predicting the majority class can achieve high accuracy while detecting none of the important cases. Report class-specific rates and decision loss.
What does an ROC curve show?
It shows the sensitivity and false-positive-rate trade-off across possible cutoffs. It evaluates ranking behaviour, not probability calibration or the economic value of one operating point.
How should a cutoff be selected?
Compare validated confusion counts and expected consequences under realistic prevalence and capacity. The best cutoff can change when error costs, base rates or operational limits change.
How is precision different from sensitivity?
Sensitivity asks what fraction of actual positives were detected. Precision asks what fraction of predicted positives were truly positive, so it changes when prevalence changes even if class-conditional performance stays similar.
Can capacity be part of threshold selection?
Yes. If only a fixed number of alerts can be reviewed, compare thresholds that satisfy that constraint and choose among them using expected value, loss and detection priorities on held-out data.
Exam move
Start every classification metric question by writing which outcome is positive, then draw the two-by-two table and fill TP, FN, FP and TN. Derive sensitivity, specificity and the two error rates from those counts instead of memorising disconnected formulas. Create a rare-event example to show why accuracy can hide failure.
Add a loss matrix and calculate expected cost for two thresholds, including review costs and missed-event costs on separate lines. Draw an ROC curve, mark two operating points and explain the direction of movement when the cutoff falls. Distinguish AUC, which summarises ranking, from calibration and expected business value.
Finish with a threshold recommendation that states prevalence, costs, capacity and the held-out sample used for validation. That closing move converts metric computation into predictive analytics.
Working through Classification Loss and ROC Analysis in QBUS2820? Sia is AskSia’s AI Statistics tutor — ask any QBUS2820 Classification Loss and ROC Analysis question and get a clear, step-by-step explanation grounded in how QBUS2820 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
