The Hong Kong University of Science and Technology · FACULTY OF ARTS & HUMANITIES

MAIE5103 Chap.3 Group Fairness Criteria and the Confusion Matrix

- one subject, every graph, every model, every mark
8 Chapters4-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 3 of 14 · MAIE5103

Group Fairness Criteria and the Confusion Matrix

Every criterion is a statement about rates

The group fairness criteria taught in this course all come from the same object: a confusion matrix built separately for each group. From it you compute the true-positive rate among people who should receive the good outcome, the false-positive rate among people who should not, precision among those selected, and the selection rate among everyone.

The lecture insists on one matrix per group because a single pooled matrix can look healthy while hiding the gap that matters, and because accuracy blends all of these rates with each group's base rate.

Four criteria and what each protects

Demographic parity asks for equal selection rates, which is easy to explain and useful when the true label is contested, but it ignores who is qualified.

Equal opportunity, from Hardt, Price and Srebro, asks that qualified people in every group are selected at the same rate. Equalized odds adds the same requirement for unqualified people, which usually needs group-specific thresholds. Predictive parity asks that a positive decision means the same thing in every group.

The Beyond Bias lecture groups these into independence, separation and sufficiency, and warns that none is a universal ethical certificate.

Beyond binary decisions

Calibration moves the question from a single threshold to every score level, so that a score of 0.7 means a 70% chance in every group.

Individual fairness asks that similar people be treated similarly, which depends on a similarity measure someone has to choose. Causal fairness asks whether the decision would change for the same person if the attribute changed, and needs a causal graph. Rankings need yet another measure, each group's share of exposure by position.

In this chapter

What this chapter covers

  • 01

    Building one confusion matrix per group

  • 02

    True-positive rate, false-positive rate, precision and selection

  • 03

    Why accuracy cannot settle a fairness question

  • 04

    Demographic parity and when it is used

  • 05

    Equal opportunity and equalized odds

  • 06

    Predictive parity and calibration

  • 07

    Independence, separation and sufficiency

  • 08

    Individual, causal and exposure-based fairness

Worked example · free

Check a scholarship screen against three criteria

Q [6 marks]. AskSia-authored practice. A scholarship screen is tested on two groups of 100 applicants whose eligibility was later verified. Group P: 30 eligible applicants were selected and 10 were not; 12 ineligible applicants were selected and 48 were not. Group Q: 14 eligible applicants were selected and 6 were not; 16 ineligible applicants were selected and 64 were not. Compute the rates and decide which of demographic parity, equal opportunity and predictive parity hold. The marks shown are an AskSia study allocation, not a University marking scheme.
Two groups of 200 with the same number selected.Two groups of 200 with the same number selected. Rows are the true label and columns the decision; darker cells hold more people, and Group B’s heavy false-positive cell is where its lower precision comes from.Group A, N=200TP 60FN 20FP 30TN 90Group B, N=200TP 36FN 24FP 54TN 86qualifiednot qualified
Two groups of 200 with the same number selected. Rows are the true label and columns the decision; darker cells hold more people, and Group B’s heavy false-positive cell is where its lower precision comes from.
  • 2Compute selection rates and test demographic parity.
  • 2Compute true-positive rates and test equal opportunity.
  • 2Compute precision and test predictive parity.
Selection is 42/100 = 42% for P and 30/100 = 30% for Q, so demographic parity fails. True-positive rates are 30/40 = 75% for P and 14/20 = 70% for Q, a five-point gap, so equal opportunity fails narrowly, and with groups this small that gap would need an interval before anyone called it a finding. Precision is 30/42, about 71.4%, for P and 14/30, about 46.7%, for Q, so predictive parity fails clearly: a selected Q applicant is much less likely to be eligible. Part of every gap traces to the different eligibility rates, 40% in P and 20% in Q.
Sia tip — Say the denominator out loud before you divide: eligible applicants for the true-positive rate, selected applicants for precision, everyone for selection.
Glossary

Key terms

True-Positive Rate
The share of people whose true outcome is positive who receive the positive decision, computed as TP divided by TP plus FN.
False-Positive Rate
The share of people whose true outcome is negative who nevertheless receive the positive decision, computed as FP divided by FP plus TN.
Selection Rate
The share of everyone in a group who receives the positive decision, the quantity that demographic parity equalises.
Individual Fairness
The requirement that people who are similar for the task are treated similarly, which depends entirely on how similarity is defined.
Counterfactual Fairness
The requirement that a decision would be the same for the same person if their protected attribute were different, tested against a causal graph.
Share of Exposure
A ranking measure of how much attention each group receives across positions, used because a ranked list has no single accept or reject decision.
Sufficiency
The family of criteria under which the true outcome is independent of group given the decision, of which equal precision is the positive-decision part.
FAQ

Group Fairness Criteria and the Confusion Matrix FAQ

What is the difference between equal opportunity and equalized odds?

Equal opportunity compares only people who truly deserve the favourable outcome and asks that each group's qualified members receive it at the same rate, which equalises true-positive rates. Equalized odds also asks that unqualified people are wrongly selected at the same rate in every group, so it equalises false-positive rates too. It is stricter and usually needs a different threshold for each group.

How do I compute precision from a confusion matrix?

Divide the true positives by everyone who received the positive decision, that is TP divided by TP plus FP, separately for each group. Precision answers the question of how likely a selected person is to be qualified. It changes with the base rate even when error rates are identical, which is why it can differ between groups whose true-positive and false-positive rates match.

When is demographic parity the right criterion?

The lecture lists outreach, advertising and candidate slates, and settings where the true label is unobserved or contested. Its weakness is that it ignores who is qualified: where base rates differ, forcing equal selection can mean selecting unqualified people or rejecting qualified ones, and in some credit settings in the United States it would amount to an unlawful quota.

Study strategy

Assessment move

Make yourself a blank pair of two-by-two matrices and fill them from three different invented examples until you can compute all four rates for both groups in under five minutes. Then, for each example, say which criteria hold and which harm each failure measures. The calculation is the easy part; naming what the gap means for a person is where answers earn their marks.

Working through Group Fairness Criteria and the Confusion Matrix in MAIE5103? Sia is AskSia’s AI Arts and Humanities tutor — ask any MAIE5103 Group Fairness Criteria and the Confusion Matrix question and get a clear, step-by-step explanation grounded in how MAIE5103 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 13 of your subjects at The Hong Kong University of Science and Technology, plus 1,000+ more Bibles across the AskSia library.
Sia - your MAIE5103 tutor, unlimited, worked the way the exam marks it
The full 4-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works