The Hong Kong Polytechnic University · FACULTY OF COMPUTER SCIENCE

COMP5577 Chap.2 Threat Models, Security Goals and Metrics

- one subject, every graph, every model, every mark
6 Chapters4-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 2 of 6 · COMP5577

Threat Models, Security Goals and Metrics

Five ingredients make risk testable

A threat model names the asset, adversary, capability, objective and constraint. AI assets include data, labels, weights, prompts, embeddings, memory, tools, logs and trust.

Capability must separate access from knowledge, because query-only, partial-knowledge and white-box attackers require different evaluations.

Metrics encode the attacked population

Attack success rate, clean accuracy, detector rates, precision and query budget answer different questions.

The safest calculation habit is to write each denominator in words before using the formula, then interpret the number against the stated security goal and constraint.

Separate knowledge from modification

A black-box adversary may query outputs, a gray-box adversary may also know the model family or training distribution, and a white-box adversary may have weights, gradients, prompts or code.

Those knowledge levels are separate from what can be changed. A passive observer can still infer private information, while a malicious contributor can alter data without knowing final weights. State both axes so the defense is tested against the right power.

Use constraints to close the model

Query limits, poison budgets, perturbation sizes, time, money and stealth requirements define which attempts count.

They also decide whether a result transfers. A detector evaluated on obvious outliers may fail against clean-label poison. A defense tested under a fixed query budget may fail when retries are unrestricted. The final interpretation should name the tested constraint and the first stronger condition that remains unresolved, allowing another analyst to reproduce or challenge the claim.

In this chapter

What this chapter covers

  • 01

    Assets and adversaries

  • 02

    Capability, access and knowledge

  • 03

    Objectives and attack constraints

  • 04

    Confidentiality, privacy, integrity and robustness

  • 05

    Attack success and clean accuracy

  • 06

    Detector rates, precision and query budgets

Worked example · free

Complete a retrieval threat model

Q [10 marks]. A contributor edits one indexed document to steer a procurement summary while lacking access to weights or prompts. Write the five ingredients and one success metric. The marks are an AskSia study allocation, not a University marking scheme.
The five ingredients constrain one another.The five ingredients constrain one another. The arrows show why a change in attacker access, budget or objective can turn the same asset into a different security test.A complete claim connects five threat-model ingredientsAssetAdversaryCapabilityConstraintObjective
The five ingredients constrain one another. The arrows show why a change in attacker access, budget or objective can turn the same asset into a different security test.
  • 3Name the protected decision and adversary.
  • 4State access, knowledge, objective and constraint.
  • 3Choose a metric on hostile documents.
The asset is decision integrity; the adversary is a content contributor with input-modification access and black-box knowledge. The objective is one targeted false summary under the constraint of one text document. Measure the fraction of hostile documents that cause the target false conclusion, alongside false alarms on clean documents.
Sia tip — If the metric does not change when the stated attack succeeds, the metric belongs to a different threat model.
Glossary

Key terms

Threat Model
A structured description of the asset, adversary, capability, objective and constraint behind an attack.
Black-box Access
Query access that reveals outputs without exposing weights, gradients, prompts or code.
Attack Success Rate
The fraction of attempted attacks that achieve the attacker’s objective.
False-positive Rate
The fraction of benign cases that a detector incorrectly flags as attacks.
FAQ

Threat Models, Security Goals and Metrics FAQ

Why are asset and security goal not the same thing?

The asset is what has value, such as training data, model behavior or a decision process. The security goal states the property sought under attack, such as confidentiality, integrity or availability. One asset can need several properties, and one attack can harm more than one goal.

What is the difference between access and knowledge?

Access identifies where the attacker interacts, such as data, an API, prompts, tools or logs. Knowledge identifies what the attacker knows about architecture, weights, gradients or distributions. A query-only attacker may still know the model family, so capability needs both dimensions.

How do I avoid mixing detector metrics?

Write the denominator first. True-positive rate starts with attacks, false-positive rate starts with benign inputs and precision starts with everything flagged. The same counts appear in several formulas, so a numerical answer without the population name is easy to misinterpret.

Study strategy

Exam move

Build a threat-model card for every practice case before calculating. Include the attack budget and knowledge level, then write each metric as a sentence about a population. Recalculate detector examples until true-positive rate, false-positive rate and precision no longer blur together.

Change one ingredient at a time: give a black-box attacker gradients, double the query budget, or replace observation access with data modification. State which defense claim breaks and which metric must change. Keep a denominator ledger beside your calculations and circle the population before substituting numbers. When comparing controls, apply operational constraints first, then rank the remaining security results.

End each answer with the first stronger adversary that remains unresolved. That final boundary converts a numerical result into a reproducible security claim. Use a small confusion matrix to test whether a high detection rate can coexist with poor precision when attacks are rare. Recalculate after changing only the base rate, then explain why the detector itself did not improve.

Finally, translate every fraction back into affected requests or users. Concrete counts reveal denominator mistakes and show the operational burden created by false alarms. Quote the deployment population whenever you report a detector rate.

Working through Threat Models, Security Goals and Metrics in COMP5577? Sia is AskSia’s AI Computer Science tutor — ask any COMP5577 Threat Models, Security Goals and Metrics question and get a clear, step-by-step explanation grounded in how COMP5577 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 24 of your subjects at The Hong Kong Polytechnic University, plus 1,000+ more Bibles across the AskSia library.
Sia - your COMP5577 tutor, unlimited, worked the way the exam marks it
The full 4-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works