MAIE5103 Chap.3 Group Fairness Criteria and the Confusion Matrix
Group Fairness Criteria and the Confusion Matrix
Every criterion is a statement about rates
The group fairness criteria taught in this course all come from the same object: a confusion matrix built separately for each group. From it you compute the true-positive rate among people who should receive the good outcome, the false-positive rate among people who should not, precision among those selected, and the selection rate among everyone.
The lecture insists on one matrix per group because a single pooled matrix can look healthy while hiding the gap that matters, and because accuracy blends all of these rates with each group's base rate.
Four criteria and what each protects
Demographic parity asks for equal selection rates, which is easy to explain and useful when the true label is contested, but it ignores who is qualified.
Equal opportunity, from Hardt, Price and Srebro, asks that qualified people in every group are selected at the same rate. Equalized odds adds the same requirement for unqualified people, which usually needs group-specific thresholds. Predictive parity asks that a positive decision means the same thing in every group.
The Beyond Bias lecture groups these into independence, separation and sufficiency, and warns that none is a universal ethical certificate.
Beyond binary decisions
Calibration moves the question from a single threshold to every score level, so that a score of 0.7 means a 70% chance in every group.
Individual fairness asks that similar people be treated similarly, which depends on a similarity measure someone has to choose. Causal fairness asks whether the decision would change for the same person if the attribute changed, and needs a causal graph. Rankings need yet another measure, each group's share of exposure by position.
What this chapter covers
- 01
Building one confusion matrix per group
- 02
True-positive rate, false-positive rate, precision and selection
- 03
Why accuracy cannot settle a fairness question
- 04
Demographic parity and when it is used
- 05
Equal opportunity and equalized odds
- 06
Predictive parity and calibration
- 07
Independence, separation and sufficiency
- 08
Individual, causal and exposure-based fairness
Worked example · free
Check a scholarship screen against three criteria
- 2Compute selection rates and test demographic parity.
- 2Compute true-positive rates and test equal opportunity.
- 2Compute precision and test predictive parity.
Key terms
- True-Positive Rate
- The share of people whose true outcome is positive who receive the positive decision, computed as TP divided by TP plus FN.
- False-Positive Rate
- The share of people whose true outcome is negative who nevertheless receive the positive decision, computed as FP divided by FP plus TN.
- Selection Rate
- The share of everyone in a group who receives the positive decision, the quantity that demographic parity equalises.
- Individual Fairness
- The requirement that people who are similar for the task are treated similarly, which depends entirely on how similarity is defined.
- Counterfactual Fairness
- The requirement that a decision would be the same for the same person if their protected attribute were different, tested against a causal graph.
- Share of Exposure
- A ranking measure of how much attention each group receives across positions, used because a ranked list has no single accept or reject decision.
- Sufficiency
- The family of criteria under which the true outcome is independent of group given the decision, of which equal precision is the positive-decision part.
Group Fairness Criteria and the Confusion Matrix FAQ
What is the difference between equal opportunity and equalized odds?
Equal opportunity compares only people who truly deserve the favourable outcome and asks that each group's qualified members receive it at the same rate, which equalises true-positive rates. Equalized odds also asks that unqualified people are wrongly selected at the same rate in every group, so it equalises false-positive rates too. It is stricter and usually needs a different threshold for each group.
How do I compute precision from a confusion matrix?
Divide the true positives by everyone who received the positive decision, that is TP divided by TP plus FP, separately for each group. Precision answers the question of how likely a selected person is to be qualified. It changes with the base rate even when error rates are identical, which is why it can differ between groups whose true-positive and false-positive rates match.
When is demographic parity the right criterion?
The lecture lists outreach, advertising and candidate slates, and settings where the true label is unobserved or contested. Its weakness is that it ignores who is qualified: where base rates differ, forcing equal selection can mean selecting unqualified people or rejecting qualified ones, and in some credit settings in the United States it would amount to an unlawful quota.
Assessment move
Make yourself a blank pair of two-by-two matrices and fill them from three different invented examples until you can compute all four rates for both groups in under five minutes. Then, for each example, say which criteria hold and which harm each failure measures. The calculation is the easy part; naming what the gap means for a person is where answers earn their marks.
Working through Group Fairness Criteria and the Confusion Matrix in MAIE5103? Sia is AskSia’s AI Arts and Humanities tutor — ask any MAIE5103 Group Fairness Criteria and the Confusion Matrix question and get a clear, step-by-step explanation grounded in how MAIE5103 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.