COMP9414 Chap.1 AI as Evidence and Capability
AI as Evidence and Capability
The course history uses apparent autonomy, conversational behaviour, bounded game success, embodied agents and combinations of learning with search to ask a harder question than whether a demonstration looks impressive. This chapter separates observed performance from mechanism, task boundary and broader capability claims.
It gives you a reusable way to recognise what evidence supports, what remains an inference and which changed condition would test the claim.
What this chapter covers
- 01
Observed behaviour versus an interpretation about intelligence
- 02
Mechanism evidence from search, learning or integrated agent design
- 03
Bounded success in a precisely defined task
- 04
Embodied agents facing perceptual noise and other agents
- 05
Combined methods such as learning with tree search
- 06
Boundary tests that change context, assumptions or task distribution
Worked example · free
Audit a campus-kiosk capability claim
- +1State the observed performance without adding a mental-state claim: the tested orientation responses were correct.
- +1Identify the mechanism evidence needed, such as retrieval, map search, pattern matching or generated language.
- +1Change one condition with an ambiguous room, a closed corridor or a dependent follow-up question.
- +1Conclude that the evidence supports the tested orientation task while broader understanding depends on the changed-condition results.
Key terms
- Capability claim
- A statement about what a system can do within a specified task, setting and boundary.
- Mechanism evidence
- Information showing which representation, search, learning or control process produced an observed result.
- Boundary test
- A changed-condition evaluation designed to reveal where a capability claim stops holding.
- Task distribution
- The collection of cases or situations from which evaluation examples are drawn.
- Transfer
- Successful performance after the task, context or assumptions differ from the original evaluation.
AI as Evidence and Capability FAQ
Why is a convincing demonstration not enough?
It establishes behaviour in the observed setting, but a broader claim also needs evidence about the mechanism, repeatability and task boundary. Changing wording, state or operating conditions can reveal whether the behaviour transfers or depends on a narrow setup.
How should I compare historical AI milestones?
Compare the task model, mechanism and evaluation conditions rather than arranging systems on one vague intelligence scale. A bounded game search, an embodied team agent and a system combining learning with search support different claims because they face different observations and actions.
What makes a capability conclusion testable?
Name the behaviour, the setting in which it was measured and one condition that could weaken the inference. A conclusion becomes useful when another reader can reproduce the observation and understand which new result would justify revising it.
Exam move
Build a four-column table headed observation, mechanism, boundary and judgement. Place each course milestone in one row and write only what the teaching material supports. Then turn a headline about an AI system into a bounded claim by specifying the task and evaluation. Invent one changed condition that targets the broadest unsupported word, such as any, understand or autonomous.
Practise explaining why failure on a relevant boundary test narrows a claim without erasing the original success. For timed answers, use one milestone as evidence and a contrasting milestone to show how a richer environment changes the capability being tested. Close with the evidence that would support the broader interpretation.
This prepares the reasoning habit used throughout the guide: algorithms and plans also earn guarantees only under explicit assumptions. Add a second pass that audits causal language. Circle every sentence that moves from performance to an explanation and write the evidence needed for that move in the margin. If the mechanism is unknown, use conditional wording rather than filling the gap with a familiar technology label.
Next create a transfer matrix with task wording, environment, feedback and evaluation on the columns. Change one column at a time and predict whether the original result should persist. This makes boundary tests diagnostic rather than merely difficult. Finally, compare two systems that reach the same outcome through different methods and explain why an equal score cannot decide which will adapt better.
Rehearse the conclusion aloud in twenty seconds: supported behaviour, operating scope, open mechanism question and decisive next test. That compact sequence is easier to reproduce than a chronological list and gives every historical example an analytical purpose. For a final retrieval drill, hide the milestone names and reconstruct each case from its task boundary, mechanism clue and strongest defensible conclusion.
Then swap the boundary between two cases and explain why the inference changes. This forces you to recall the analytical role of each example instead of relying on a familiar name. Mark any conclusion that lacks a possible disconfirming result and rewrite it until the evidence could genuinely change your judgement.
Working through AI as Evidence and Capability in COMP9414? Sia is AskSia’s AI Computer Science tutor — ask any COMP9414 AI as Evidence and Capability question and get a clear, step-by-step explanation grounded in how COMP9414 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
