FIT5202 · Data Processing for Big Data
Machine Learning: Classification
Week 6 of Monash FIT5202 covers classification: decision trees built with the ID3 splitting criterion (entropy and information gain, log base 10 as taught), pruning and maxDepth, random forests and boosting, parallel/distributed tree learning, and classifier evaluation with the confusion matrix (accuracy, precision, recall, F1). Entropy-and-information-gain splits and precision/recall/F1 calculations are among the most reliable written-response items in the exam and central to Assignment 2 Part A.
What this chapter covers
- 01Classification = predicting discrete categories (spam/not, churn/not, fraud/not)
- 02Decision-tree terminology: root, splitting, decision node, leaf, pruning, branch, homogeneity
- 03Entropy H(S) = Σ pᵢ·log(1/pᵢ) as a measure of uncertainty (log base 10 in the taught examples)
- 04Information gain IG(S,A) = H(S) − Σ (|Sᵢ|/|S|)·H(Sᵢ); split on the largest gain (ID3)
- 05maxDepth and overfitting; pruning to generalise
- 06Random forest (majority vote), bagging (bootstrap aggregating) and gradient boosting
- 07Parallel classification: data parallelism (vertical) vs result parallelism (horizontal tree nodes)
- 08Evaluation: confusion matrix → accuracy, precision, recall, F1
Evaluating a classifier from its confusion matrix
- +1Accuracy = (TP + TN) / total = (40 + 45) / 100 = 85 / 100 = 0.85 (85% of all emails classified correctly).
- +1Precision = TP / (TP + FP) = 40 / (40 + 10) = 40 / 50 = 0.80. Of everything flagged as spam, 80% really was spam.
- +1Recall = TP / (TP + FN) = 40 / (40 + 5) = 40 / 45 ≈ 0.889. Of all real spam, 88.9% was caught.
- +1F1 = 2·P·R / (P + R) = 2 × 0.80 × 0.889 / (0.80 + 0.889) = 1.422 / 1.689 ≈ 0.842, the harmonic mean balancing precision and recall.
- +1Interpretation: accuracy (0.85) can hide class imbalance; precision, recall and F1 expose the false-positive versus false-negative trade-off — here recall exceeds precision, so the classifier catches most spam but at the cost of some false alarms.
Key terms
- Entropy H(S)
- A measure of uncertainty in a set: H(S) = Σ pᵢ·log(1/pᵢ). Zero for a pure (single-class) set, maximal for a perfectly mixed one. FIT5202 worked examples use log base 10.
- Information gain
- The reduction in entropy from a split: IG(S,A) = H(S) − Σ (|Sᵢ|/|S|)·H(Sᵢ). ID3 chooses the attribute with the largest information gain at each node.
- ID3
- The Iterative Dichotomiser 3 algorithm: compute the dataset entropy, then for each attribute compute the weighted entropy and information gain, split on the largest gain, and recurse until leaves are homogeneous.
- Random forest
- An ensemble of decision trees, each trained on a different bootstrap sample and feature subset; it outputs the majority vote of the trees, reducing the high variance of a single tree.
- Precision / Recall / F1
- Precision = TP/(TP+FP) (correctness of positive predictions); recall = TP/(TP+FN) (coverage of actual positives); F1 = 2·P·R/(P+R) is their harmonic mean. Use these instead of accuracy on imbalanced data.
- Overfitting / maxDepth
- Growing a tree to full depth on training data memorises noise and generalises poorly. maxDepth (the longest root-to-leaf path) limits growth; pruning removes sub-nodes to simplify the model.
Machine Learning: Classification FAQ
Why does FIT5202 use log base 10 for entropy?
Because the taught worked examples were computed in base 10 (for example H(9 Yes, 5 No) ≈ 0.283), so reproducing those numbers requires base 10. Information theory often uses base 2 (bits), but for this unit's exam use base 10 so your entropy and gain values match the marking scheme. State the base you use.
How do I choose the attribute to split on in a decision tree?
Compute the dataset entropy, then for each candidate attribute compute the weighted entropy of its branches and the information gain IG = H(S) − weighted entropy. Split on the attribute with the largest information gain, because it removes the most uncertainty. Repeat at each child until the branches are pure (ID3).
When should I report precision and recall instead of accuracy?
Whenever the classes are imbalanced or the costs of false positives and false negatives differ. Accuracy can look high just by predicting the majority class, whereas precision and recall expose how well the model handles the positive class, and F1 balances them. Spam, fraud and disease detection are classic cases.
Can AI help me with decision trees and evaluation?
Yes. Sia can walk an entropy-and-information-gain split line by line, build a confusion matrix into precision/recall/F1, and quiz you on overfitting and pruning — explaining the method and checking your reasoning. It does not complete graded assessment for you, and Monash academic-integrity rules apply.
Exam move
Split Week 6 into two drills. First, the ID3 split: dataset entropy, per-branch entropy, weighted entropy, information gain, pick the largest — practised in log base 10 until the numbers come out clean. Second, evaluation: from any confusion matrix, produce accuracy, precision, recall and F1 with a line of interpretation, remembering FP sits under precision and FN under recall. These are among the most predictable Q2–Q6 items, so rehearse both end to end on Moodle problems through SWOTVAC, and use Sia to generate fresh trees and matrices to solve.
Working through Machine Learning: Classification in FIT5202? Sia is AskSia’s AI Information Technology tutor — ask any FIT5202 Machine Learning: Classification question and get a clear, step-by-step explanation grounded in how FIT5202 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
