PHIL7002 Chap.2 Opacity, Transparency and Explainability
Opacity, Transparency and Explainability
Right answers for the wrong reasons
A horse called Clever Hans appeared to do arithmetic, tapping out answers with roughly eighty-nine per cent accuracy, until investigation revealed he was reading his owner's unconscious cues. The chapter is a set of variations on that sentence.
Modern equivalents are easy to find: a classifier that separates wolves from huskies by the snow in the background, a boat detector that needs visible water, a dumbbell detector that needs a visible arm.
Shortcuts of this kind produce errors precisely when context shifts, and high-stakes decisions are rarely standard cases.
Accuracy is a starting point, not a summary
Accuracy reports the share of predictions that were correct and says nothing about who the errors fall on.
A confusion matrix separates false positives from false negatives, and choosing between them is a value judgement in the shape of a threshold: a system that flags everyone has no false negatives, one that flags nobody has no false positives. The course puts both poles plainly, noting that you might prefer false positives when selecting for a rehabilitation programme and false negatives when deciding guilt.
Precision, recall, calibration, subgroup performance, reliability, stability and robustness each report something accuracy cannot, and drift comes in two kinds, one where the inputs shift and one where the target itself changes.
Six senses of one word
Transparency covers at least six distinct things: political answerability, responsibility and liability, inspectability and auditability, accessibility, intelligibility and justifiability.
Intelligibility is explicitly relative to an audience and a purpose, which is why the recurring question is meaningful to whom. Engineers want features, thresholds and logs; lawyers want compliance evidence and decision pathways; clinicians want relevant factors and uncertainty; citizens want reasons and a route to recourse.
Process transparency asks a further set of questions that require no access to the model at all: who built it, on what terms, who benefits, who bears the risk, what alternatives were rejected, and how a user obtains human review.
Testing an explanation instead of trusting it
A model that is interpretable by design uses the same function to predict and to explain, so faithfulness is not a separate question.
A post-hoc explanation is a model of a model, and the course's own analogy is a transit map: useful for some purposes and not the territory. Local surrogate methods can be unstable enough that small sampling changes alter the explanation, and the stronger charge is that many post-hoc explanations deliver good-sounding reasons that are not the true reasons, which can close an inquiry that should have continued.
Language models make the point sharply: they can be steered to an answer by a factor they never mention while producing a confident account of a different route to it.
The standards dispute, and the law around it
Should machines face a higher transparency standard than the humans they replace?
The case for parity runs from human opacity, through the fact that people are the comparison being used, to the cost of blocking beneficial technology. Critics answer with four asymmetries: scale, the impossibility of contestation, the gap between explanation and accountability, and the fact that explanation is not justification.
A separate challenge attacks the parity argument's premise instead, holding that interpretable models routinely match black-box performance on structured data, so using an unexplainable system in a high-stakes setting is a choice rather than a necessity. The regulatory picture is plural, and whether an individual right to an explanation exists is genuinely contested.
What this chapter covers
- 01
Clever Hans and modern shortcut learning
- 02
Beyond accuracy: precision, recall, calibration, subgroup performance
- 03
The confusion matrix and domain-dependent error preference
- 04
Six senses of transparency, and meaningful to whom
- 05
Process transparency, and ex ante against ex post explanation
- 06
Interpretable by design against post-hoc surrogate
- 07
Ersatz understanding and the fidelity test
- 08
The double-standards argument and the challenge to its premise
- 09
Risk-tiered law, and the right to explanation dispute
Route one complaint to the sense of transparency that can answer it
- 3Name the sense the applicant needs, and the evidence that would satisfy it.
- 3Name the sense the regulator needs, and why it need not be intelligible to her.
- 3Name the sense the journalist needs, and the cost of confusing the three.
Key terms
- Shortcut Learning
- A pattern in which a model reaches correct outputs through a feature that is incidental to the task, such as a snowy background, so that performance collapses when context shifts.
- Confusion Matrix
- A table separating correct and incorrect predictions of each class, which makes the trade-off between false positives and false negatives explicit.
- Precision
- The share of flagged cases that were genuine, which matters most where a false alarm is costly.
- Recall
- The share of genuine cases that were flagged, which matters most where a miss is costly.
- Process Transparency
- Openness about how a system came to be deployed: who built it, on what terms, who benefits, which alternatives were rejected and how a user obtains human review.
- Ersatz Understanding
- The impression of insight produced by a persuasive post-hoc explanation that is not in fact the model's reason, which can close an inquiry rather than advance it.
- Faithfulness
- The property of an explanation that it reports the factors that actually drove an output, tested by removing those factors and seeing whether the output changes.
- Concept Drift
- A change in the thing being predicted rather than in the input distribution, which is why monitoring needs a retraining threshold rather than a periodic review.
Opacity, Transparency and Explainability FAQ
Why is it not enough to report that a model is highly accurate?
Because accuracy is silent about who absorbs the errors, and a system can improve on aggregate while concentrating its mistakes on one group. The measures that fill the gap are precision and recall, which separate the two error types, calibration, which asks whether a stated probability matches outcomes, and subgroup performance, which asks whether the whole pattern differs between groups.
A defence of a deployment that rests on accuracy alone has not yet addressed the objection that is usually being made.
What is the difference between an interpretable model and an explained one?
An interpretable model uses the same function to make the prediction and to give the reason, so there is no gap between them to verify. An explained model is a black box with a second, simpler model built afterwards to approximate it, which means the explanation is a description of a description.
The practical consequence is that only the second arrangement owes a fidelity test, and the course's stronger claim is that where an interpretable model performs comparably, choosing the black box is a decision that needs defending.
Does the law give an individual the right to an explanation of an automated decision?
It is disputed. One reading holds that the relevant European provisions require that a person be told something substantive about how a decision works and what it may lead to, in advance, which is a right to be informed rather than a right to an explanation afterwards.
A second holds that those provisions, read together with the right to contest a decision and obtain human intervention, only make sense if an individualised explanation is owed. A third suggests the dispute matters less than it appears, since even a bare duty to inform creates genuine obligations ahead of time.
Exam move
Take one system you use and write the same complaint about it six times, once in each sense of transparency, then write what evidence would end each version. The exercise is quick and it builds the reflex the chapter is really testing, which is naming the sense before assessing the disclosure. Follow it by running the three-move fidelity test on any explanation feature the system offers you.
Working through Opacity, Transparency and Explainability in PHIL7002? Sia is AskSia’s AI Arts and Humanities tutor — ask any PHIL7002 Opacity, Transparency and Explainability question and get a clear, step-by-step explanation grounded in how PHIL7002 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
