The Hong Kong University of Science and Technology · FACULTY OF ARTS & HUMANITIES

MAIE5103 Chap.12 Human Oversight and Handover Failures

- one subject, every graph, every model, every mark
8 Chapters4-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 12 of 14 · MAIE5103

Human Oversight and Handover Failures

A person in the loop is not automatically a control

The Tempe collision is the lecture's main case. A developmental automated driving system detected an object 5.6 seconds before impact but never identified the object as a pedestrian or forecast where she was going, judged a crash imminent at 1.2 seconds, and then withheld braking for a second without alerting its single operator.

That operator was combining road monitoring with event tagging, the company rarely used its ability to review operators, and the factory collision-avoidance braking had been disabled without a full replacement.

The lecture's conclusion is that a human fallback protects anyone only if watching is possible, taking over works, a backup layer remains and someone checks that oversight happens.

Facts, findings and theory

The investigation's probable cause was the operator's distraction by a personal phone, with weaknesses in the company's risk assessment, its supervision of operators, its measures against complacency and its safety culture listed as contributing.

The lecture asks students to keep three things apart: the recorded facts, the investigator's determinations, and later recommendations.

It also warns against renaming findings with theory labels, such as diagnosing a cognitive bias the report never diagnosed.

Handover, comparators and clinical evidence

Bainbridge's irony of automation, that automating routine work leaves people the hardest recoveries, frames Air France 447, read from the investigator's final report as a multi-factor accident involving inconsistent airspeed data, an autopilot disconnection and an unrecognised stall.

A meta-analysis of 106 experiments shows that human and AI teams beat humans alone on average but not the better of the two, so the comparator decides the claim. The MASAI screening trial then shows how to read a randomised interim result narrowly.

In this chapter

What this chapter covers

  • 01

    The Tempe braking handover, second by second

  • 02

    Why one operator was not a safety net

  • 03

    Probable cause, contributing factors and control boundaries

  • 04

    Keeping findings apart from theory labels

  • 05

    Bainbridge's irony of automation and AF447

  • 06

    Signal integrity, mode transition and recovery

  • 07

    Choosing the right comparator for human and AI teams

  • 08

    Appraising the MASAI interim analysis

Worked example · free

Assign findings to control boundaries

Q [6 marks]. AskSia-authored practice. An investigation into a warehouse robot collision finds that the robot's obstacle sensor misread reflective tape, that the supervisor responsible for stopping robots was also loading trucks, and that the company had never reviewed its stop-button logs. Assign each finding to a control boundary and name the evidence needed to assess it. The marks shown are an AskSia study allocation, not a University marking scheme.
The last six seconds on the recorded timeline.The last six seconds on the recorded timeline. Detection came early, the imminent-crash judgement late, and the shaded second of withheld braking carried no alert, so the human fallback had less time than the axis suggests.braking withheld, no alert-6 s-4 s-2 s0 sobject detectedcrash judged imminentsuppression endslate steering
The last six seconds on the recorded timeline. Detection came early, the imminent-crash judgement late, and the shaded second of withheld braking carried no alert, so the human fallback had less time than the axis suggests.
  • 2Assign the sensor finding and name its evidence.
  • 2Assign the supervisor finding and name its evidence.
  • 2Assign the log finding and name its evidence.
The sensor misreading sits at the design boundary of the system and robot; assessing it needs the sensor's test record on reflective surfaces and the decision to deploy without one. The supervisor's dual role sits at the monitoring boundary, but the more important question is about safety assurance: who approved combining stopping duties with loading, and on what analysis. The unreviewed logs sit at the safety-assurance boundary of the company, and the evidence needed is its oversight procedures and why they were not used. Placing each finding on a boundary prevents the common error of blaming the nearest person for an organisational design.
Sia tip — Write the boundary name in brackets after each finding you cite: monitoring, safety assurance or design; it stops you blaming one layer for another's failure.
Glossary

Key terms

Irony of Automation
Bainbridge's observation that automating routine work leaves people the hardest tasks of diagnosis and recovery when automation hands control back.
Mode Transition
The moment control and diagnosis shift from an automated system to people, as when an autopilot disconnects.
Probable Cause
An investigator's formal determination of the main cause of an accident, distinct from contributing factors and later recommendations.
Control Boundary
The layer at which a safety control sits, such as operator monitoring, organisational safety assurance or system design.
Comparator
The baseline a result is measured against, such as humans alone or the better of human or AI alone, which decides what a hybrid result means.
Interim Analysis
A pre-planned look at a trial's data before its final outcome is known, used to decide whether continuing is safe rather than to prove superiority.
FAQ

Human Oversight and Handover Failures FAQ

What caused the Uber self-driving car crash in Tempe?

The investigator's probable cause was the operator's failure to monitor the road and the automated system while visually distracted by a personal phone. Contributing factors included the company's inadequate safety-risk assessment, ineffective oversight of operators, inadequate mechanisms against complacency and its safety culture.

The record also shows the system withheld braking for a second without an alert and that factory emergency braking had been disabled.

What is the irony of automation?

It is Bainbridge's 1983 argument that automation takes over routine work and leaves people with the hardest part, diagnosing problems and recovering control when the automation hands back. The lecture uses Air France 447 to illustrate it, as a multi-factor accident in which inconsistent airspeed data, an autopilot disconnection and an unrecognised stall combined with training, interfaces and crew coordination.

Does combining humans and AI always improve performance?

Not against the right comparator. A meta-analysis of 106 experiments and 370 effect sizes found that human and AI teams outperformed humans alone on average, but the pooled effect against the better of humans alone or AI alone was negative, and results varied widely by task. Any claim that a hybrid works should say which comparison it is making.

Study strategy

Assessment move

For each case, make a three-column note: recorded fact, formal determination, later recommendation. Then add the control boundary for each finding. Practising this separation is the best preparation for a case study analysis on safety, because it is the structure the lecture uses and the one that keeps an analysis from overstating what an investigation found.

Working through Human Oversight and Handover Failures in MAIE5103? Sia is AskSia’s AI Arts and Humanities tutor — ask any MAIE5103 Human Oversight and Handover Failures question and get a clear, step-by-step explanation grounded in how MAIE5103 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 13 of your subjects at The Hong Kong University of Science and Technology, plus 1,000+ more Bibles across the AskSia library.
Sia - your MAIE5103 tutor, unlimited, worked the way the exam marks it
The full 4-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works