COMP90087 · The Ethics of Artificial Intelligence
Trust, Power, and Accountability
Week 7 asks whether AI’s harms are distinctive or just “more of the same,” answering with five mechanisms of harm — datafication, automation, scalability, transformative effects and inscrutability/opacity — illustrated by Weapons of Math Destruction, Automating Inequality and Seeing Like a State. It then teaches Nissenbaum’s four barriers to accountability, an exam-core list you must be able to name and apply. Together they explain who can be held responsible when a scaled AI system causes harm.
What this chapter covers
- 01The five features that make AI morally concerning: datafication, automation, scalability, transformative effects, inscrutability/opacity
- 02Datafication — abstracting human systems into simplified, fallible measurements that can reshape the system to fit the abstraction
- 03Scalability — automation lets harms amplify rapidly before detection; concentrates undemocratic power in system owners
- 04Anchor cases: O’Neil (teacher value-added scoring), Eubanks (automated welfare eligibility), Scott (the forestry parable)
- 05What accountability is (Nissenbaum): blameworthy/praiseworthy, liable to sanction, able to give an account, responsible for rectifying
- 06Nissenbaum’s four barriers (exam-core): the problem of many hands, treating bugs as inevitable, the computer as scapegoat, ownership without liability
- 07The Therac-25 case as an illustration of the problem of many hands
- 08Mitigations: an explicit standard of care, distinguishing accountability from liability, strict/producer liability
Diagnose an accountability gap with Nissenbaum’s four barriers
- +1Problem of many hands. The vendor, the agency, the data suppliers and the caseworkers all contributed, so the causal antecedents don’t converge with a single decision-maker — no one person is obviously answerable, and the structure may be arranged to diffuse blame.
- +1Computer as scapegoat. “The system flagged it” offloads responsibility onto the machine, which cannot perform the social act of giving an account — the humans use it as a target to evade responsibility onto.
- +1Ownership without liability, plus bugs-as-inevitable. The vendor keeps the profit of ownership while the licence disclaimer sheds the liability, and treating the data-matching error as an unavoidable “bug” excuses the engineers from blame.
- +1Mitigation. Impose an explicit standard of care and distinguish accountability from liability — e.g. strict producer responsibility and a clear line of who must foresee, prevent and rectify — so “I couldn’t have known” is not a blanket excuse.
Key terms
- Datafication
- Abstracting complex human systems into simplified, fallible measurements so machines can process them; it can destroy information and reshape the system to fit the abstraction, enabling disparate impacts (O’Neil, Scott).
- Scalability (as a harm mechanism)
- Automation lets a system scale rapidly, amplifying harms before detection or safeguards, detaching feedback, and concentrating undemocratic power in the system’s owners.
- Accountability (Nissenbaum)
- Being answerable for an outcome: morally blameworthy or praiseworthy, liable to sanction or restitution, able to give an account of what was done, and responsible for rectifying errors.
- Problem of many hands
- Nissenbaum’s first barrier: so many actors and technologies contribute that the causal antecedents don’t converge with the locus of decision-making, and blame becomes hard to assign (illustrated by Therac-25).
- Computer as scapegoat
- Nissenbaum’s barrier in which the machine appears to occupy the “accountable party” role, letting humans offload responsibility — even though a computer cannot perform the social activities of accountability.
- Ownership without liability
- Nissenbaum’s barrier where law lags so the positive aspects of ownership (praise, profit) are decoupled from the negative (blame, liability), often via software licences and disclaimers.
Trust, Power, and Accountability FAQ
What are the five mechanisms of harm and why do they matter?
Datafication, automation, scalability, transformative effects and inscrutability/opacity. They answer the week’s framing question — whether AI’s harms are distinctive or just more of the same — by showing how AI amplifies and reshapes harm (e.g. scaling a biased scoring model across many districts before anyone can intervene). Know them as a named set and tie each to one of the anchor books (O’Neil, Eubanks, Scott).
How do I answer a Nissenbaum accountability question?
Name the relevant barrier(s) from the four — many hands, bugs treated as inevitable, computer as scapegoat, ownership without liability — and tie each to a concrete feature of the scenario, then propose a mitigation (standard of care, distinguishing accountability from liability, strict producer responsibility). The exam rewards naming the barrier and showing where it operates, not just defining accountability.
Why is the Therac-25 case used here?
Because its radiation overdoses traced to coding errors, faulty hardware, poor testing, exaggerated reliability claims and inadequate training all at once — the textbook illustration of the “problem of many hands,” where responsibility is spread across so many actors that no one is obviously accountable. It is a safe, citable public case for a short answer.
Can AI help me learn the accountability material for COMP90087?
Yes. Sia can quiz you on the five harm mechanisms and Nissenbaum’s four barriers, set fresh scenarios where you diagnose which barriers operate, and check your mitigations — explaining each step. It mirrors how the University of Melbourne assesses this and does not do graded work for you; the subject’s integrity and GenAI rules apply.
Exam move
Commit two named lists to recall: the five mechanisms of harm and Nissenbaum’s four barriers, each pinned to a case (O’Neil / Eubanks / Scott for the mechanisms; Therac-25 for many hands). Practise the diagnostic move — given a scenario, name which barriers operate and tie each to a specific feature — because that is exactly the short-answer format. Learn the three mitigations (standard of care, accountability vs liability, strict producer responsibility) so you can always close with a remedy. This is exam-core material and it recurs when you analyse governance and generative AI, so make it solid ahead of the closed-book hurdle exam.
Working through Trust, Power, and Accountability in COMP90087? Sia is AskSia’s AI AI Ethics tutor — ask any COMP90087 Trust, Power, and Accountability question and get a clear, step-by-step explanation grounded in how COMP90087 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
