IS6400 Chap.5 Decision Trees and Ensemble Learning
Decision Trees and Ensemble Learning
A tree partitions with questions
Each split seeks child nodes that are less mixed in the target than their parent. Gini impurity is zero for a pure node and highest for balanced binary classes.
Candidate splits must use child-size weights, because a tiny pure leaf cannot compensate for leaving most observations mixed.
Ensembles trade one path for stability
Bagging and random forests vary data and features to reduce dependence on one unstable tree. Boosting corrects earlier errors sequentially and can reduce bias while becoming sensitive to noise.
Validation, threshold cost and monitoring decide whether the complexity is worthwhile.
Audit a split from parent to children
Compute the parent impurity, each child impurity and the child-size-weighted total before finding the reduction. Equal averaging is wrong when the children contain different numbers of rows. A large local reduction supports that split only on the training sample.
Check minimum leaf size, repeated validation and whether a small branch represents enough cases to support a stable operational rule.
Know what creates ensemble diversity
Bagging fits learners to varied resamples and averages or votes, primarily reducing variance. Random forests also vary the feature candidates so one dominant predictor does not create nearly identical trees.
Boosting fits learners sequentially to correct earlier errors and can reduce bias, but it may chase noise. Compare candidates on the same split and threshold, not by training purity or the visual complexity of a tree.
Separate prediction from policy
A tree or ensemble supplies a class score or probability estimate; the operating threshold determines action and workload.
Inspect calibration by score band and errors by segment, then price false positives and false negatives at the live threshold. Retain a shallow baseline beside a stronger ensemble so maintenance cost and drift are visible. After launch, monitor score distributions, leaf or segment traffic and labelled outcomes before deciding whether a shift requires retraining, threshold revision or data repair.
What this chapter covers
- 01
Recursive feature splits
- 02
Gini impurity and weighted reduction
- 03
Depth and minimum leaf controls
- 04
Pruning against validation evidence
- 05
Bagging and random feature subsets
- 06
Boosting and threshold-specific evaluation
Worked example · free
Compare parent and weighted child impurity
- 2Find parent impurity. This mark allocation is not an official University marking scheme.
- 4Find both child impurities and size-weight them.
- 3Find the reduction and comment on stability.
Key terms
- Tree Split
- A feature rule dividing rows into child nodes to reduce target mixture.
- Gini Impurity
- One minus the sum of squared class proportions inside a node.
- Pruning
- Removing or preventing branches whose training detail does not justify their validation performance.
- Bagging
- Fitting models on varied resamples and combining them to reduce prediction variance.
- Boosting
- Building learners sequentially so later learners focus more strongly on earlier errors.
Decision Trees and Ensemble Learning FAQ
Why can a decision tree overfit even when every split looks sensible?
Repeated splits can isolate tiny training-specific groups and capture noise. Control depth and leaf size, prune using validation evidence, and inspect whether the first rules remain stable across folds or time windows.
What does a random forest gain over one tree?
Resampling rows and varying candidate features create diverse trees whose errors can average out. The forest is usually more stable, but the production decision no longer follows one complete readable path.
When might the smaller tree be preferable?
A smaller tree can win when performance is close, decisions require a consistent explanation, and the organisation cannot maintain complex local explanations and drift monitoring. Selection should include operating burden, not only test score.
Exam move
Compute Gini for a balanced node, a nearly pure node and a pure node. Work one split with child-size weights, then state the reduction. On a separate card compare bagging, random forests and boosting by how diversity is created and which error tendency is targeted. Finish every tree answer by distinguishing training purity from validation stability and by naming the threshold-specific business cost.
Work a second candidate split so the comparison includes two weighted child impurities rather than one isolated calculation. Then draw how maximum depth, minimum leaf size and pruning change the same tree. For ensembles, state whether diversity comes from rows, features or sequential error correction.
End with a threshold table showing confusion counts, workload and calibration, and explain when the smaller tree should win because maintenance and explanation outweigh a marginal validation gain. Revisit the same case after a feature disappears at deployment and decide whether the tree can still be scored, must be retrained, or should fall back to the baseline.