MGMT90280 · Managerial Decision Analytics

Predictive Data Mining: Classification Trees & Performance Measures

Predictive data mining is the Week-9 topic of MGMT90280 Managerial Decision Analytics at the University of Melbourne, and the supervised-learning counterpart to the subject's descriptive (clustering / association) mining. It covers train / validation / test partitioning and overfitting, the confusion matrix and its measures (overall and class error rates, sensitivity, specificity, precision, F1), lift and ROC / AUC for ranking models, and classification trees — splitting on impurity, pruning on the validation set, and reading each root-to-leaf path as an if–then rule. It is the final-exam Q5 material, built and scored in Analytic Solver Basic.

What this chapter covers

• 01Supervised learning: predict a known class label; the three-way data partition train (build) / validation (tune & guard overfitting) / test (one unbiased final score)
• 02Overfitting: training error keeps falling toward 0 while validation error turns up — a large train-vs-validation gap is the symptom
• 03Confusion matrix (Class 1 = positive): TN n₀₀, FP n₀₁, FN n₁₀, TP n₁₁, with actual down the rows and predicted across the columns
• 04Overall error rate (n₁₀+n₀₁)/all and accuracy = 1 − overall error; class-1 error n₁₀/actual-1 and class-0 error n₀₁/actual-0
• 05Sensitivity (recall) n₁₁/(n₁₁+n₁₀), specificity n₀₀/(n₀₀+n₀₁), precision n₁₁/(n₁₁+n₀₁), F1 = harmonic mean of precision and recall
• 06The cutoff dial (default 0.5): raising it lowers class-1 hits (↑ class-1 error, ↓ class-0 error) and vice versa
• 07Lift: decile lift = decile response rate ÷ overall rate; cumulative lift curve bowing above the diagonal beats random targeting
• 08ROC curve (sensitivity vs 1 − specificity) and AUC: 0.5 = random, 1.0 = perfect, larger AUC = better model choice
• 09Classification trees: split on impurity, predict a node's majority class, prune on the validation set, read root-to-leaf paths as if–then rules
• 10Which measure to optimise: pick by the costlier error — FN worse → sensitivity, FP worse → precision, both matter → F1

A loan-default classifier: every error rate and measure, plus an overfitting check

Key terms

Training / validation / test partition

The three disjoint slices of the data: the training set builds the models, the validation set tunes them and guards against overfitting, and the test set is used once at the very end for an unbiased performance estimate.

Overfitting

When a model fits the training data much better than new data. The symptom is a large gap between a low training error and a higher validation / test error; the cure is a simpler model (e.g. a pruned tree).

Confusion matrix

A 2×2 table of counts for a classifier with actual class down the rows and predicted class across the columns: TN (n00), FP (n01), FN (n10), TP (n11). Every performance measure reads off these four cells.

Sensitivity (recall)

n11/(n11+n10): of the cases that truly are positive, the fraction the model catches. Equals 1 − the class-1 error rate; optimise it when a missed positive (false negative) is the costly error.

Specificity

n00/(n00+n01): of the cases that truly are negative, the fraction correctly cleared. Equals 1 − the class-0 error rate; it is recall for the negative class.

Precision

n11/(n11+n01): of the cases the model flags as positive, the fraction that truly are positive. Optimise it when a false positive (acting on a wrong flag) is the costly error.

ROC curve and AUC

The ROC plots sensitivity against 1 − specificity as the cutoff sweeps; the AUC (area under it) summarises ranking quality — 0.5 is random, 1.0 is perfect, and a larger AUC means a better model.

Classification tree (pruning)

A model that recursively splits data on one variable to reduce impurity; a leaf predicts its majority class. Because training error only falls as the tree grows, the tree is pruned back on the validation set to the best-pruned size.

Predictive Data Mining: Classification Trees & Performance Measures FAQ

Exam move

Anchor everything on the confusion matrix: memorise the four cells (TN, FP, FN, TP with actual down the rows, predicted across the columns) and derive every measure from them rather than rote-learning formulas. Drill the denominators until they are automatic — overall error over all cases, class errors over each actual class, recall over actual positives, precision over predicted positives — because swapping recall's and precision's denominators is the marker's favourite slip. Practise the 'which measure' logic (missed positive costly → sensitivity; false alarm costly → precision; both matter → F1) and always pair a number with a sentence of interpretation. For trees, be able to read a diagram into majority-class predictions, node and tree error rates, and a plain-English rule, and remember to judge tree size by validation error, not training error. This is the final-exam Q5 material and it is fast, high-yield arithmetic, so bank those marks and leave time to interpret. Budget time in proportion to marks (about 1.2 minutes per mark, since the exam is 2 hours of writing for 100 marks), and confirm all assessment dates and rules on the LMS.

Working through Predictive Data Mining: Classification Trees & Performance Measures in MGMT90280? Sia is AskSia’s AI Statistics tutor — ask any MGMT90280 Predictive Data Mining: Classification Trees & Performance Measures question and get a clear, step-by-step explanation grounded in how MGMT90280 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.