COMP5577 Chap.2 Threat Models, Security Goals and Metrics
Threat Models, Security Goals and Metrics
Five ingredients make risk testable
A threat model names the asset, adversary, capability, objective and constraint. AI assets include data, labels, weights, prompts, embeddings, memory, tools, logs and trust.
Capability must separate access from knowledge, because query-only, partial-knowledge and white-box attackers require different evaluations.
Metrics encode the attacked population
Attack success rate, clean accuracy, detector rates, precision and query budget answer different questions.
The safest calculation habit is to write each denominator in words before using the formula, then interpret the number against the stated security goal and constraint.
Separate knowledge from modification
A black-box adversary may query outputs, a gray-box adversary may also know the model family or training distribution, and a white-box adversary may have weights, gradients, prompts or code.
Those knowledge levels are separate from what can be changed. A passive observer can still infer private information, while a malicious contributor can alter data without knowing final weights. State both axes so the defense is tested against the right power.
Use constraints to close the model
Query limits, poison budgets, perturbation sizes, time, money and stealth requirements define which attempts count.
They also decide whether a result transfers. A detector evaluated on obvious outliers may fail against clean-label poison. A defense tested under a fixed query budget may fail when retries are unrestricted. The final interpretation should name the tested constraint and the first stronger condition that remains unresolved, allowing another analyst to reproduce or challenge the claim.
What this chapter covers
- 01
Assets and adversaries
- 02
Capability, access and knowledge
- 03
Objectives and attack constraints
- 04
Confidentiality, privacy, integrity and robustness
- 05
Attack success and clean accuracy
- 06
Detector rates, precision and query budgets
Worked example · free
Complete a retrieval threat model
- 3Name the protected decision and adversary.
- 4State access, knowledge, objective and constraint.
- 3Choose a metric on hostile documents.
Key terms
- Threat Model
- A structured description of the asset, adversary, capability, objective and constraint behind an attack.
- Black-box Access
- Query access that reveals outputs without exposing weights, gradients, prompts or code.
- Attack Success Rate
- The fraction of attempted attacks that achieve the attacker’s objective.
- False-positive Rate
- The fraction of benign cases that a detector incorrectly flags as attacks.
Threat Models, Security Goals and Metrics FAQ
Why are asset and security goal not the same thing?
The asset is what has value, such as training data, model behavior or a decision process. The security goal states the property sought under attack, such as confidentiality, integrity or availability. One asset can need several properties, and one attack can harm more than one goal.
What is the difference between access and knowledge?
Access identifies where the attacker interacts, such as data, an API, prompts, tools or logs. Knowledge identifies what the attacker knows about architecture, weights, gradients or distributions. A query-only attacker may still know the model family, so capability needs both dimensions.
How do I avoid mixing detector metrics?
Write the denominator first. True-positive rate starts with attacks, false-positive rate starts with benign inputs and precision starts with everything flagged. The same counts appear in several formulas, so a numerical answer without the population name is easy to misinterpret.
Exam move
Build a threat-model card for every practice case before calculating. Include the attack budget and knowledge level, then write each metric as a sentence about a population. Recalculate detector examples until true-positive rate, false-positive rate and precision no longer blur together.
Change one ingredient at a time: give a black-box attacker gradients, double the query budget, or replace observation access with data modification. State which defense claim breaks and which metric must change. Keep a denominator ledger beside your calculations and circle the population before substituting numbers. When comparing controls, apply operational constraints first, then rank the remaining security results.
End each answer with the first stronger adversary that remains unresolved. That final boundary converts a numerical result into a reproducible security claim. Use a small confusion matrix to test whether a high detection rate can coexist with poor precision when attacks are rare. Recalculate after changing only the base rate, then explain why the detector itself did not improve.
Finally, translate every fraction back into affected requests or users. Concrete counts reveal denominator mistakes and show the operational burden created by false alarms. Quote the deployment population whenever you report a detector rate.
Working through Threat Models, Security Goals and Metrics in COMP5577? Sia is AskSia’s AI Computer Science tutor — ask any COMP5577 Threat Models, Security Goals and Metrics question and get a clear, step-by-step explanation grounded in how COMP5577 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.