The Hong Kong Polytechnic University · FACULTY OF COMPUTER SCIENCE

COMP5577 Chap.1 AI Systems and Security Attack Surfaces

- one subject, every graph, every model, every mark
6 Chapters3-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 1 of 6 · COMP5577

AI Systems and Security Attack Surfaces

Security begins before deployment

AI security covers the data, labels, training pipeline, model weights, prompts, retrieval, tools, logs and approvals that turn an input into an action. The central diagnostic move is to locate when hostile influence enters.

Training-time attacks change what the model learns; inference-time attacks change what a trained system receives or does.

Choose the right neighboring concept

Cybersecurity protects conventional infrastructure and information assets. AI safety includes harmful behavior that may occur without an attacker.

A security answer therefore needs an adversary, a capability and an objective rather than only a harmful outcome.

Follow authority across boundaries

A modern AI application joins parties that control different surfaces. A data owner decides collection and retention, a provider controls weights and training procedures, an application team assembles prompts and retrieval, and a tool operator authorises actions.

Security failures often occur where data or authority crosses between them. Mark each crossing and ask whether untrusted content is being treated as instruction, whether a supplied artifact is trusted without provenance, or whether an output can trigger an action without independent approval.

Match the control to the stage

Source verification and label review operate before learning.

Triggered evaluation and model inspection test learned behavior. Prompt separation and tool permissions operate during deployment. Rate limits and monitoring constrain repeated use. Human approval protects consequential actions. A generic recommendation to add monitoring is weak unless it states what event is recorded, who sees it and which attack signal would cause intervention.

The lifecycle makes that assignment explicit and shows where recovery must begin.

In this chapter

What this chapter covers

  • 01

    AI security and trustworthy operation

  • 02

    Training-time and inference-time boundaries

  • 03

    AI security, cybersecurity and AI safety

  • 04

    Data, model, prompt and tool attack surfaces

  • 05

    Defense parties and control ownership

  • 06

    Defense in depth across the lifecycle

Worked example · free

Place a retrieval attack on the lifecycle

Q [8 marks]. A public page injects an instruction into a retrieval assistant that can open review tickets. Classify the stage, capability and primary goal. The marks are an AskSia study allocation, not a University marking scheme.
The lifecycle divides the attack surface.The lifecycle divides the attack surface. Red marks the stages where an attacker changes learning; green marks the stages where a deployed model, prompt or tool can be manipulated.Security changes as data becomes actionDataTrainingModelDeploymentToolstraining-timeinference-time
The lifecycle divides the attack surface. Red marks the stages where an attacker changes learning; green marks the stages where a deployed model, prompt or tool can be manipulated.
  • 3Locate the changed object and lifecycle stage.
  • 3Name attacker access and the crossed trust boundary.
  • 2Assign the primary goal and defense owner.
The attack is inference-time because weights remain unchanged. The adversary modifies retrieved content and crosses the boundary from untrusted evidence to instruction. Integrity is primary because a decision is steered, and the application owner controls prompt separation and tool authorization.
Sia tip — Ask what changed: learning data points to training time, while prompts, context or tool calls point to inference time.
Glossary

Key terms

Attack Surface
A lifecycle point where hostile observation, modification or disruption can affect an AI system.
Training-time Attack
An attack that changes data, labels, code or weights before deployment.
Inference-time Attack
An attack that changes deployed inputs, context, tools or query load while weights remain fixed.
Defense Party
The person or organization that owns controls capable of blocking the stated attacker.
FAQ

AI Systems and Security Attack Surfaces FAQ

How do I decide whether an incident is an AI security attack?

Identify an adversary and show how that party observes, modifies or disrupts an AI asset or behavior. Harm caused only by a badly specified objective may be an AI safety failure instead. A conventional credential theft may be cybersecurity even when the affected application happens to use a model.

Why does the lifecycle stage matter?

The stage determines what the attacker can change and who can stop it. Poisoned training examples require source control, data inspection or retraining, while a prompt injection requires separation of untrusted content, tool authorization and runtime monitoring. Similar outcomes can therefore need different evidence and remedies.

Who is responsible for defense in a multi-party system?

Assign each control to the party that owns it. Providers protect weights and training protocols, application owners control prompts, retrieval, tools and logging, data owners govern access and retention, and human approvers own high-impact actions. A recommendation without an implementer is not an operational defense.

Study strategy

Exam move

Take a familiar AI service and draw its data, training, model, deployment and tool stages. Add one attack at each stage, then name the defense party and the evidence that would reveal it. Repeat with the same outward failure entering at a different stage, such as a wrong decision caused first by poisoned labels and then by prompt injection. Compare the controls and tests side by side.

Next mark every trust boundary where data, identity, authority or tool access changes hands. For each boundary, state what is trusted, what should remain untrusted and what log or approval would reveal misuse. Finish by rewriting broad advice such as add monitoring into an owned control with a trigger and response.

This exercise makes attack names secondary to the causal path and trains you to assign remedies to parties that can implement them. For a final recall pass, shuffle six incident cards and classify each without using the attack name printed in the notes. Require a lifecycle stage, changed object, security property, owner and observable signal on every card.

If two classifications remain plausible, write the missing fact that would separate them. This turns ambiguity into a threat-model question and exposes whether you understand the system boundary rather than merely recognising vocabulary. Check that each proposed control acts before the harmful transition it is supposed to prevent.

Working through AI Systems and Security Attack Surfaces in COMP5577? Sia is AskSia’s AI Computer Science tutor — ask any COMP5577 AI Systems and Security Attack Surfaces question and get a clear, step-by-step explanation grounded in how COMP5577 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 24 of your subjects at The Hong Kong Polytechnic University, plus 1,000+ more Bibles across the AskSia library.
Sia - your COMP5577 tutor, unlimited, worked the way the exam marks it
The full 3-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works