DATA4207 · Data Analysis in the Social Sciences
Predicting Outcomes: Interpretation, Classification and Ethics
Week 5 deepens logistic-regression interpretation — turning coefficients into predicted probabilities — and introduces classification models such as decision trees, evaluated with confusion matrices and in-sample versus out-of-sample accuracy. It confronts the ethics of prediction using recidivism/bail (COMPAS) data: identifiability, algorithmic bias and unjustified confidence in 'objective' models. This is the week the individual Research Plan (20%) is due, so the chapter also frames how to justify a chosen method.
What this chapter covers
- 01Interpreting logistic models: predicted probabilities of different outcomes from the coefficients
- 02Prediction vs causation; three conditions for causation: temporal precedence, covariation, ruling out alternatives
- 03Train/test split (commonly 80/20) and k-fold cross-validation; in-sample vs out-of-sample evaluation
- 04Overfitting — fitting training noise — and mitigations (random forests, regularisation)
- 05Decision trees: root → binary splits → leaves; interpretable, non-linear, but prone to overfitting
- 06Classification evaluation via the confusion matrix: accuracy, precision, recall, F1, ROC/AUC
- 07Ethics of prediction: algorithmic bias, black-box confidence, the COMPAS recidivism example
- 08Research Plan (20%, Week 5): justify the question, data and chosen method
Reading a classifier's confusion matrix
- +1Lay out the confusion matrix (rows = actual, columns = predicted): true positives TP = 45, false negatives FN = 15, false positives FP = 20, true negatives TN = 80 — total 160.
- +1Accuracy = (TP + TN)/total = (45 + 80)/160 = 125/160 ≈ 0.78 (about 78% of cases classified correctly).
- +1Precision = TP/(TP + FP) = 45/65 ≈ 0.69 (of those flagged positive, ~69% truly are). Recall = TP/(TP + FN) = 45/60 = 0.75 (of true positives, 75% are caught).
- +1Ethical caution: out-of-sample accuracy is the honest test, and headline accuracy hides class-specific error (here recall 0.75 means one in four positives is missed). Before using such a model for bail or similar decisions, check for algorithmic bias across groups — the COMPAS case shows 'objective' models can encode unfair, opaque outcomes.
Key terms
- Confusion matrix
- A table cross-tabulating actual against predicted classes (TP, FN, FP, TN). Accuracy, precision, recall and F1 are all computed from its cells; row-normalising gives the per-class error rates.
- Decision tree
- A supervised model that splits data on feature values through internal decision nodes to leaf predictions. Interpretable and assumption-free, it handles mixed data but is prone to overfitting — mitigated with random forests or boosting.
- In-sample vs out-of-sample
- Performance on the data used to fit the model versus on held-out data. Out-of-sample (via a test split or k-fold cross-validation) is the real test, because a model can fit training noise and look deceptively strong in-sample.
- Overfitting
- When a model captures noise in the training data and so predicts unseen data poorly. Detected by a gap between train and test accuracy and reduced with simpler models, regularisation or ensembles.
- Algorithmic bias
- Systematic unfairness in a model's predictions across groups, often hidden by a black-box model that projects false objectivity. The COMPAS recidivism tool is the unit's canonical example of why prediction ethics matter.
- Precision and recall
- Precision = TP/(TP + FP), the share of positive predictions that are correct; recall = TP/(TP + FN), the share of true positives that are caught. They trade off and matter more than raw accuracy when classes are imbalanced or costs are asymmetric.
Predicting Outcomes: Interpretation, Classification and Ethics FAQ
Why isn't accuracy enough to judge a classifier?
Accuracy hides which class the model gets wrong. With imbalanced data a model can score high accuracy by always predicting the majority class while missing most of the cases you care about. Precision and recall expose that: recall tells you how many true positives you catch, precision how trustworthy a positive prediction is. For high-stakes decisions you also weigh ROC/AUC and, crucially, fairness across groups.
What's the ethics point the unit wants me to make?
That predictive models are not neutral. Algorithmic bias can produce discriminatory outcomes, and opaque 'black-box' models create unjustified confidence in supposedly objective decisions — the COMPAS recidivism example and the Target pregnancy-prediction case are the illustrations. A good answer asks what 'fair' or 'just' means for the specific decision, whether bias can ever be fully removed, and prioritises accountability and interpretability.
How does this week connect to the Research Plan?
The Research Plan (20%) is due in Week 5, and it asks you to justify a question, dataset and analytic method. This week's material — interpreting predictions, choosing between logistic regression and a tree, and weighing ethics — is exactly the reasoning the plan wants you to demonstrate. Confirm the plan's due date and requirements on Canvas.
Can AI help me with classification in DATA4207?
Yes, as a study aid. Sia can explain confusion-matrix metrics, in-sample versus out-of-sample testing, decision trees and the prediction-ethics framing, and check your calculations and reasoning step by step. It teaches the method and checks your thinking; it does not do graded work, and University of Sydney academic-integrity rules apply — confirm on Canvas whether AI is permitted for the plan.
Assessment move
Get comfortable turning a fitted model into predictions and then judging those predictions honestly. Practise computing accuracy, precision and recall from a confusion matrix, and always split your data or cross-validate so you report out-of-sample performance, not the flattering in-sample number. Fit both a logistic regression and a decision tree on the same task to feel the interpretability/flexibility trade-off. Because the 20% Research Plan is due this week, use the ethics and method-justification framing directly: write two lines justifying why your chosen model fits your question and what could go wrong with it. Rehearse a substantive ethics paragraph rather than a generic one, and confirm the plan's requirements and due date on Canvas.
Working through Predicting Outcomes: Interpretation, Classification and Ethics in DATA4207? Sia is AskSia’s AI Statistics tutor — ask any DATA4207 Predicting Outcomes: Interpretation, Classification and Ethics question and get a clear, step-by-step explanation grounded in how DATA4207 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
