MAIE5103 Chap.12 Human Oversight and Handover Failures
Human Oversight and Handover Failures
A person in the loop is not automatically a control
The Tempe collision is the lecture's main case. A developmental automated driving system detected an object 5.6 seconds before impact but never identified the object as a pedestrian or forecast where she was going, judged a crash imminent at 1.2 seconds, and then withheld braking for a second without alerting its single operator.
That operator was combining road monitoring with event tagging, the company rarely used its ability to review operators, and the factory collision-avoidance braking had been disabled without a full replacement.
The lecture's conclusion is that a human fallback protects anyone only if watching is possible, taking over works, a backup layer remains and someone checks that oversight happens.
Facts, findings and theory
The investigation's probable cause was the operator's distraction by a personal phone, with weaknesses in the company's risk assessment, its supervision of operators, its measures against complacency and its safety culture listed as contributing.
The lecture asks students to keep three things apart: the recorded facts, the investigator's determinations, and later recommendations.
It also warns against renaming findings with theory labels, such as diagnosing a cognitive bias the report never diagnosed.
Handover, comparators and clinical evidence
Bainbridge's irony of automation, that automating routine work leaves people the hardest recoveries, frames Air France 447, read from the investigator's final report as a multi-factor accident involving inconsistent airspeed data, an autopilot disconnection and an unrecognised stall.
A meta-analysis of 106 experiments shows that human and AI teams beat humans alone on average but not the better of the two, so the comparator decides the claim. The MASAI screening trial then shows how to read a randomised interim result narrowly.
What this chapter covers
- 01
The Tempe braking handover, second by second
- 02
Why one operator was not a safety net
- 03
Probable cause, contributing factors and control boundaries
- 04
Keeping findings apart from theory labels
- 05
Bainbridge's irony of automation and AF447
- 06
Signal integrity, mode transition and recovery
- 07
Choosing the right comparator for human and AI teams
- 08
Appraising the MASAI interim analysis
Worked example · free
Assign findings to control boundaries
- 2Assign the sensor finding and name its evidence.
- 2Assign the supervisor finding and name its evidence.
- 2Assign the log finding and name its evidence.
Key terms
- Irony of Automation
- Bainbridge's observation that automating routine work leaves people the hardest tasks of diagnosis and recovery when automation hands control back.
- Mode Transition
- The moment control and diagnosis shift from an automated system to people, as when an autopilot disconnects.
- Probable Cause
- An investigator's formal determination of the main cause of an accident, distinct from contributing factors and later recommendations.
- Control Boundary
- The layer at which a safety control sits, such as operator monitoring, organisational safety assurance or system design.
- Comparator
- The baseline a result is measured against, such as humans alone or the better of human or AI alone, which decides what a hybrid result means.
- Interim Analysis
- A pre-planned look at a trial's data before its final outcome is known, used to decide whether continuing is safe rather than to prove superiority.
Human Oversight and Handover Failures FAQ
What caused the Uber self-driving car crash in Tempe?
The investigator's probable cause was the operator's failure to monitor the road and the automated system while visually distracted by a personal phone. Contributing factors included the company's inadequate safety-risk assessment, ineffective oversight of operators, inadequate mechanisms against complacency and its safety culture.
The record also shows the system withheld braking for a second without an alert and that factory emergency braking had been disabled.
What is the irony of automation?
It is Bainbridge's 1983 argument that automation takes over routine work and leaves people with the hardest part, diagnosing problems and recovering control when the automation hands back. The lecture uses Air France 447 to illustrate it, as a multi-factor accident in which inconsistent airspeed data, an autopilot disconnection and an unrecognised stall combined with training, interfaces and crew coordination.
Does combining humans and AI always improve performance?
Not against the right comparator. A meta-analysis of 106 experiments and 370 effect sizes found that human and AI teams outperformed humans alone on average, but the pooled effect against the better of humans alone or AI alone was negative, and results varied widely by task. Any claim that a hybrid works should say which comparison it is making.
Assessment move
For each case, make a three-column note: recorded fact, formal determination, later recommendation. Then add the control boundary for each finding. Practising this separation is the best preparation for a case study analysis on safety, because it is the structure the lecture uses and the one that keeps an analysis from overstating what an investigation found.
Working through Human Oversight and Handover Failures in MAIE5103? Sia is AskSia’s AI Arts and Humanities tutor — ask any MAIE5103 Human Oversight and Handover Failures question and get a clear, step-by-step explanation grounded in how MAIE5103 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.