ECON7880 Chap.3 Decision Trees and Information Gain
Decision Trees and Information Gain
Measure uncertainty with counts
A decision tree chooses questions that create purer child nodes.
This chapter calculates binary entropy from class shares, combines children using their sizes and subtracts weighted child impurity from the parent to obtain information gain.
Stop before purity becomes memorisation
Tiny pure leaves can encode individual training observations rather than repeatable structure.
Validation, depth limits, minimum leaf sizes and pruning control that risk, while path interpretation turns a fitted tree into an auditable rule.
Write node counts before probabilities
Every entropy calculation begins with class counts local to one node. Convert those counts to proportions using the node total, not the full dataset. A pure node has entropy zero, while an even binary node has one bit.
These boundary values are fast checks on arithmetic. If a purported probability is negative, exceeds one or the class shares do not sum to one, stop before calculating logarithms.
Weight children by how much data they carry
A candidate split can create one impressive tiny child and one large mixed child. Equal averaging would overstate the tiny branch.
Multiply each child entropy by its share of the parent, sum the weighted values and subtract from parent entropy.
Compare candidate gains using the same parent value and delay rounding until the end, because close splits can reverse when intermediate values are shortened.
Read a tree as a collection of conditional rules
A path to a leaf joins every split condition with “and.” Translate the path into words, then attach the leaf sample size and positive rate. That makes the coverage and evidence visible.
A leaf probability is a training proportion rather than a guaranteed future probability, so held-out results and stability across folds must accompany any operational interpretation.
Control complexity with evidence
Growing until every leaf is pure usually memorises training cases.
Compare depths, minimum leaf sizes or pruning strengths inside cross-validation and keep an untouched test set for the selected procedure. When two trees perform similarly, prefer the smaller structure if it preserves the decision-relevant metric. Monitor leaf volumes after deployment because drift can empty a once-important branch or change its observed outcome rate.
What this chapter covers
- 01
Parent class counts
- 02
Binary entropy
- 03
Weighted child entropy
- 04
Information gain
- 05
Leaf probabilities
- 06
Pruning and minimum leaves
Worked example · free
Calculate entropy reduction
- 2Calculate parent entropy.
- 4Weight both child entropies.
- 3Interpret the resulting gain.
Key terms
- Node Entropy
- Class uncertainty within one tree node, calculated from that node’s class proportions.
- Weighted Impurity
- The sum of child impurities multiplied by their shares of the parent observations.
- Pruning
- Removing weak branches to trade some training fit for simpler and more stable predictions.
Decision Trees and Information Gain FAQ
When is binary entropy highest?
It is highest at one bit when the two classes are equally represented. It falls toward zero as one class dominates and equals zero at a pure node with only one class remaining.
Why are child entropies weighted?
A large child represents more of the parent observations than a small child. Equal averaging would let a tiny branch influence the split as much as a large one.
Is a pure leaf always desirable?
No. Purity based on very few observations may be unstable. Report leaf size and validate pruning or minimum-size choices on unseen data, then inspect whether the same path and class rate recur across folds or later samples. Stable larger leaves may be more useful for monitoring and deployment decisions.
Exam move
Recalculate two candidate splits by hand. Keep counts, proportions, entropies, child weights and gain on separate lines, then compare the results before rounding. Translate one leaf path into a business rule with its sample size and class rate. Next, imagine that the last split creates leaves of one observation each and decide whether validation supports pruning.
Finish by explaining why a pure leaf, high training accuracy and high information gain are local facts rather than automatic proof of generalisation. Draw a second tree with the same training accuracy but fewer leaves and compare validation recall, stability and interpretability. Check entropy boundary cases for a pure node and an even node without a calculator.
Finally, inspect the root split across several folds and write what a changing variable or threshold says about the strength of the evidence. Double every node count and confirm that gain is unchanged, then explain why reliability still depends strongly on sample size.
Working through Decision Trees and Information Gain in ECON7880? Sia is AskSia’s AI Economics tutor — ask any ECON7880 Decision Trees and Information Gain question and get a clear, step-by-step explanation grounded in how ECON7880 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
