COMP5577 AI Security
COMP5577 Overview
- The Hong Kong Polytechnic University
- Semester 1, 2026/27
- Postgraduate Computer Science
- Final Exam Assessment
- Canvas course materials
What this course asks you to secure
AI security is wider than protecting a model file. The captured teaching materials follow data, labels, training, weights, deployment inputs, prompts, retrieval, external tools, logs and human approvals.
- Assessed by A written assignment, an in-class test, a final project and a final exam.
- Highest weight The published final exam carries seventy per cent of the course grade.
- Core method State the threat model, choose a matching metric, then test the defense against that attacker.
- Where to confirm Check Canvas for the final exam format, current logistics and any assessment changes.
How COMP5577 is assessed
| Component | Weight | Format |
|---|---|---|
| Written Assignment | 10% | One written assignment |
| In-class Test | 10% | One in-class test |
| Final Project | 10% | Code, report and presentation in a group of three or four |
| Final Exam | 70% | Final exam; format is confirmed on Canvas |
The published teaching material lists four components whose numeric weights total 100%. The captured materials name the final exam but do not publish its duration, sections, question types or permitted materials, so confirm the current format and logistics on Canvas.
Assessment structure
The segment widths reproduce the published weights and total 100%.
Current COMP5577 dates
| Date | Item | Control |
|---|---|---|
| 30 October 2026 | Written Assignment | Assignment 1 due at 23:59 in the captured course site. |
Dates are as published in the captured Canvas syllabus and Assignment 1 brief. Confirm exact deadlines and submission settings in the live LMS.
What COMP5577 covers
The guide follows the captured teaching sequence from AI attack surfaces and threat models through poisoning, backdoor detection, evasion and privacy attacks.
AI Systems and Security Attack Surfaces
AI security against cybersecurity and safety, the AI lifecycle, training-time and inference-time attack surfacesSecurity begins before deployment
AI security covers the data, labels, training pipeline, model weights, prompts, retrieval, tools, logs and approvals that turn an input into an action.
- AI security and trustworthy operation
- Training-time and inference-time boundaries
- AI security, cybersecurity and AI safety
- Data, model, prompt and tool attack surfaces
- Defense parties and control ownership
- Defense in depth across the lifecycle
Threat Models, Security Goals and Metrics
assets, adversaries, capabilities, objectives, constraints, security goals, attack and detector metricsFive ingredients make risk testable
A threat model names the asset, adversary, capability, objective and constraint. AI assets include data, labels, weights, prompts, embeddings, memory, tools, logs and trust.
- Assets and adversaries
- Capability, access and knowledge
- Objectives and attack constraints
- Confidentiality, privacy, integrity and robustness
- Attack success and clean accuracy
- Detector rates, precision and query budgets
Data Poisoning and Backdoor Attacks
untargeted and targeted poison, label flipping, trigger-conditioned behavior and supply-chain riskPoisoning changes the learned rule
Data poisoning enters before deployment through examples, labels, feedback or model weights. Untargeted attacks lower broad utility, while targeted attacks change a selected outcome.
- Data contributors and training access
- Untargeted and targeted poisoning
- Label-flipping attacks
- Backdoor triggers and target classes
- Clean accuracy and triggered attack success
- Model poisoning and checkpoint provenance
Detecting and Repairing Poisoned Models
prevention, activation clustering, poison confirmation, model repair and triggered reevaluationDefense has four different jobs
Prevention controls sources, detection finds suspicious structure, robustness limits damage and recovery restores a usable model.
- Prevention, detection, robustness and recovery
- Outliers and anomaly assumptions
- K-means and density-based clustering
- Activation vectors within classes
- Trigger inspection and confirmation
- Repair, retraining and reevaluation
Evasion Attacks and Robust Defenses
FGSM and PGD, perturbation budgets, sanitization, adversarial training and certificationEvasion changes inputs after training
Adversarial examples exploit the geometry of a fixed model. FGSM takes one step in the sign of the input gradient for a non-targeted objective, while PGD repeats projected steps within a perturbation region.
- Inference-time evasion
- FGSM gradient-sign updates
- PGD and targeted objectives
- Input sanitization and detection
- Adversarial training and trade-offs
- Empirical evaluation versus certification
Privacy Attacks and Model Extraction
membership inference, inversion, memorized data, substitute models and privacy defensesOutputs can disclose without a database breach
Membership inference asks whether a named record was used. Attribute inference predicts a hidden property.
- Membership inference and shadow models
- Overfitting and generalization gaps
- Attribute inference
- White-box and black-box inversion
- Training-data memorization and extraction
- Model extraction and interface controls
Each surface gives an adversary a different capability, so the first task in any answer is to state where hostile influence enters and which party controls that boundary.
The reasoning pattern behind the topics
The course repeatedly joins a threat model to a metric. Assets, adversaries, capabilities, objectives and constraints turn a vague risk into a testable claim.
Clean accuracy, attack success rate, detector rates, poisoning rate and adversarial accuracy then measure different populations. Strong ordinary performance does not prove security under attack, because an evaluation that omits the attacker’s condition never exercises the failure.
Training-time and inference-time attacks
Data poisoning and backdoors change what a model learns before deployment.
Evasion changes inputs after training. Privacy attacks use outputs to infer membership, attributes or memorised content, while model extraction uses queries to copy behaviour.
The defenses differ because the attacker’s access differs: provenance and data inspection address training inputs, adversarial evaluation addresses deployed predictions, and output controls or privacy-aware training address disclosure.
How to prepare from the evidence available
The official materials publish the assessment weights but do not give the captured final exam’s duration, sections, question types or permitted materials.
This guide therefore does not invent a paper format. It builds original practice around the analytical moves demonstrated by the teaching and written assignment: classify the attack, show calculations, interpret the denominator, compare defenses under constraints and state the limitation of the evidence.
Worked example · free
Take one FGSM step and verify the prediction
- 2Compute the original score and class.
- 3Apply the signed perturbation to both features.
- 3Recompute the score and state whether the class changed.
Key terms
- Threat Model
- A structured account of the asset, adversary, capability, objective and constraint that make an attack claim testable.
- Attack Surface
- A point in the AI lifecycle where an adversary can observe, modify or disrupt data, models, context or actions.
- Attack Success Rate
- The fraction of attempted attacks that achieve the attacker’s stated objective on the attacked population.
- Clean Accuracy
- The fraction of ordinary clean inputs classified correctly, which does not by itself measure behavior under attack.
- Data Poisoning
- A training-time attack that changes examples, labels or feedback so the learned behavior serves an adversarial goal.
- Backdoor
- A trigger-conditioned behavior learned during training that preserves ordinary predictions while mapping triggered inputs to an attacker-chosen result.
- Activation Clustering
- A detection method that groups internal representations within each class to identify samples learned through a different feature pattern.
- Evasion Attack
- An inference-time attack that changes a deployed model’s input to change its prediction without retraining the model.
- Adversarial Training
- Training on both benign and adversarial examples so known perturbations are incorporated into the learning objective.
- Certified Robustness
- A formal guarantee that a stated property holds for every allowed perturbation in a defined region under explicit assumptions.
- Membership Inference
- An attack that estimates whether a named record was included in a model’s training data.
- Model Extraction
- An attack that uses queries or outputs to reproduce a model’s behavior or recover a substitute model.
COMP5577 FAQ
How is this course assessed?
The published split is a written assignment worth ten per cent, an in-class test worth ten per cent, a final project worth ten per cent and a final exam worth seventy per cent. The project includes code, a report and a presentation completed in a small group. Check Canvas for current submission and examination logistics.
What is known about the final exam format?
The captured teaching material names the final exam and its seventy per cent weight, but it does not state the duration, sections, question types or permitted materials. This guide does not infer those details from the written assignment. Confirm the current format on Canvas when the University publishes it.
What makes a threat model complete?
Name the asset to protect, the adversary, what that adversary can observe or modify, the outcome they want and the constraint on the attempt. For AI systems, also state the lifecycle stage, attacker knowledge, trust boundary, primary security goal and the party that owns the proposed control.
Why does high clean accuracy not prove robustness?
Clean accuracy measures ordinary inputs, while an attack activates a different condition. A backdoor can preserve clean predictions and still send most trigger-bearing inputs to an attacker-chosen class. Robust evaluation must exercise the stated adversary, attack budget and trigger or perturbation rather than reuse only the clean test distribution.
How do poisoning and evasion differ?
Poisoning acts during training by changing data, labels, feedback or model weights, so it alters what the model learns. Evasion acts after training by changing a deployed input while weights stay fixed. The distinction changes the attacker capability, the evidence required and the party that can implement the defense.
What does a strong defense answer include?
It connects the control to an adversary capability, reports clean and attacked performance on the right populations, and states the threat-model limits. Detection should lead to confirmation and repair, followed by reevaluation. Empirical resistance to selected attacks should not be described as certified robustness.
How are privacy attacks kept distinct?
Membership inference asks whether a record was used, attribute inference predicts a hidden characteristic, inversion reconstructs representative or sensitive features, training-data extraction recovers memorised examples and model extraction copies behavior. Describe exactly what the attacker recovered before naming the attack or selecting a mitigation.
Are there past papers in the available materials?
No past examination or sample final is present in the captured course materials. The available assessment artifact is the current written assignment. Practice here is independently authored from the concepts, calculation types and reasoning expectations shown by the teaching materials, not copied or lightly reworded from University questions.
Questions students ask in each COMP5577 chapter
- How do I decide whether an incident is an AI security attack?
- Identify an adversary and show how that party observes, modifies or disrupts an AI asset or behavior. Harm caused only by a badly specified objective may be an AI safety failure instead. A conventional credential theft may be cybersecurity even when the affected application happens to use a model. Chapter 01 · AI Systems and Security Attack Surfaces
- Why are asset and security goal not the same thing?
- The asset is what has value, such as training data, model behavior or a decision process. The security goal states the property sought under attack, such as confidentiality, integrity or availability. One asset can need several properties, and one attack can harm more than one goal. Chapter 02 · Threat Models, Security Goals and Metrics
- How does targeted poisoning differ from a backdoor?
- Targeted poisoning changes a selected case or subgroup while preserving broad performance. A backdoor adds a secret trigger condition that causes an attacker-chosen output at inference time. Every backdoor is targeted poisoning in this framing, but not every targeted poisoning attack uses a trigger. Chapter 03 · Data Poisoning and Backdoor Attacks
- Why is a suspicious cluster not enough to delete data?
- Clustering supplies geometric separation, not semantic meaning. Legitimate accents, devices, writing styles or subgroups can form a minority cluster. Inspect examples, labels, triggers and predictions before removal or a security defense may create a utility or fairness failure. Chapter 04 · Detecting and Repairing Poisoned Models
- Why is FGSM called a white-box attack?
- Its standard form needs the gradient of loss with respect to the input, which requires knowledge of the model and differentiation path. A transfer attack may use a substitute model, but the gradient still comes from a model the attacker can inspect. Chapter 05 · Evasion Attacks and Robust Defenses
- Does a reconstructed face prove that exact image was in training?
- Not necessarily. Model inversion may recover a prototype or features that strongly activate a class without reproducing a specific record. Call it training-data extraction only when the evidence supports exact or near-exact recovery of memorized material. Chapter 06 · Privacy Attacks and Model Extraction
How to study for the exam
Start every revision problem with a one-line threat model before reaching for a formula. Write the asset, adversary, access, knowledge, objective, constraint, stage, primary security goal and defense party. Then write the denominator in words and only then calculate.
Keep a separate column for limitations: clean evaluation omits attack conditions, a detector threshold trades misses against false alarms, adversarial training depends on attacks and budgets, and a formal certificate holds only under its assumptions. Work original cases without notes, compare your chain with the completed solution, and correct the first step where your reasoning diverged.
In the final week, practise switching between attack names that look similar: targeted poisoning versus backdoors, privacy versus confidentiality, membership versus inversion, and empirical robustness versus certification. Confirm the final exam format and logistics on Canvas rather than planning around an inferred paper structure.
Your AI Computer Science tutor for COMP5577
Stuck on a hard COMP5577 question? Sia is AskSia’s AI Computer Science tutor — ask any COMP5577 AI Security question and get a clear, step-by-step explanation grounded in how the course is actually taught and assessed. Read this whole study guide free, then take your hardest questions to Sia.