City University of Hong Kong · FACULTY OF INFORMATION TECHNOLOGY

IS6400 Chap.5 Decision Trees and Ensemble Learning

- one subject, every graph, every model, every mark
6 Chapters3-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 5 of 10 · IS6400

Decision Trees and Ensemble Learning

A tree partitions with questions

Each split seeks child nodes that are less mixed in the target than their parent. Gini impurity is zero for a pure node and highest for balanced binary classes.

Candidate splits must use child-size weights, because a tiny pure leaf cannot compensate for leaving most observations mixed.

Ensembles trade one path for stability

Bagging and random forests vary data and features to reduce dependence on one unstable tree. Boosting corrects earlier errors sequentially and can reduce bias while becoming sensitive to noise.

Validation, threshold cost and monitoring decide whether the complexity is worthwhile.

Audit a split from parent to children

Compute the parent impurity, each child impurity and the child-size-weighted total before finding the reduction. Equal averaging is wrong when the children contain different numbers of rows. A large local reduction supports that split only on the training sample.

Check minimum leaf size, repeated validation and whether a small branch represents enough cases to support a stable operational rule.

Know what creates ensemble diversity

Bagging fits learners to varied resamples and averages or votes, primarily reducing variance. Random forests also vary the feature candidates so one dominant predictor does not create nearly identical trees.

Boosting fits learners sequentially to correct earlier errors and can reduce bias, but it may chase noise. Compare candidates on the same split and threshold, not by training purity or the visual complexity of a tree.

Separate prediction from policy

A tree or ensemble supplies a class score or probability estimate; the operating threshold determines action and workload.

Inspect calibration by score band and errors by segment, then price false positives and false negatives at the live threshold. Retain a shallow baseline beside a stronger ensemble so maintenance cost and drift are visible. After launch, monitor score distributions, leaf or segment traffic and labelled outcomes before deciding whether a shift requires retraining, threshold revision or data repair.

In this chapter

What this chapter covers

  • 01

    Recursive feature splits

  • 02

    Gini impurity and weighted reduction

  • 03

    Depth and minimum leaf controls

  • 04

    Pruning against validation evidence

  • 05

    Bagging and random feature subsets

  • 06

    Boosting and threshold-specific evaluation

Worked example · free

Compare parent and weighted child impurity

Q [9 marks]. AskSia-authored practice. The listed marks are AskSia study allocations, not a University rubric. A parent has five positive and five negative rows. A split produces a six-row child with five positive and a four-row child with all four negative. Compute Gini reduction.
The left path explains one decision.The left path explains one decision. The right side shows three varied trees voting two to one, reducing reliance on any single unstable split.One readable tree becomes one vote inside an ensemblelate payments > 2?low riskhigh riskTree 1: HTree 2: LTree 3: Hvote: high risk
The left path explains one decision. The right side shows three varied trees voting two to one, reducing reliance on any single unstable split.
  • 2Find parent impurity. This mark allocation is not an official University marking scheme.
  • 4Find both child impurities and size-weight them.
  • 3Find the reduction and comment on stability.
Parent Gini is 1 - (0.5 squared + 0.5 squared) = 0.5. The six-row child has Gini 1 - ((5/6) squared + (1/6) squared) = 10/36. The four-row pure child has Gini zero. Weighted child impurity is (6/10)(10/36) + (4/10)(0) = 1/6, about 0.167. Reduction is about 0.333. The split is strong on these rows, but its value still depends on validation and adequate leaf sizes rather than training purity alone.
Sia tip — Weight child impurity by the number of rows reaching each child; an equal average changes the question.
Glossary

Key terms

Tree Split
A feature rule dividing rows into child nodes to reduce target mixture.
Gini Impurity
One minus the sum of squared class proportions inside a node.
Pruning
Removing or preventing branches whose training detail does not justify their validation performance.
Bagging
Fitting models on varied resamples and combining them to reduce prediction variance.
Boosting
Building learners sequentially so later learners focus more strongly on earlier errors.
FAQ

Decision Trees and Ensemble Learning FAQ

Why can a decision tree overfit even when every split looks sensible?

Repeated splits can isolate tiny training-specific groups and capture noise. Control depth and leaf size, prune using validation evidence, and inspect whether the first rules remain stable across folds or time windows.

What does a random forest gain over one tree?

Resampling rows and varying candidate features create diverse trees whose errors can average out. The forest is usually more stable, but the production decision no longer follows one complete readable path.

When might the smaller tree be preferable?

A smaller tree can win when performance is close, decisions require a consistent explanation, and the organisation cannot maintain complex local explanations and drift monitoring. Selection should include operating burden, not only test score.

Study strategy

Exam move

Compute Gini for a balanced node, a nearly pure node and a pure node. Work one split with child-size weights, then state the reduction. On a separate card compare bagging, random forests and boosting by how diversity is created and which error tendency is targeted. Finish every tree answer by distinguishing training purity from validation stability and by naming the threshold-specific business cost.

Work a second candidate split so the comparison includes two weighted child impurities rather than one isolated calculation. Then draw how maximum depth, minimum leaf size and pruning change the same tree. For ensembles, state whether diversity comes from rows, features or sequential error correction.

End with a threshold table showing confusion counts, workload and calibration, and explain when the smaller tree should win because maintenance and explanation outweigh a marginal validation gain. Revisit the same case after a feature disappears at deployment and decide whether the tree can still be scored, must be retrained, or should fall back to the baseline.

A+Everything unlocked
Unlocks this Bible + all 31 of your subjects at City University of Hong Kong, plus 1,000+ more Bibles across the AskSia library.
Sia - your IS6400 tutor, unlimited, worked the way the exam marks it
The full 3-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works