The Hong Kong Polytechnic University · FACULTY OF COMPUTER SCIENCE

COMP5577 Chap.6 Privacy Attacks and Model Extraction

- one subject, every graph, every model, every mark
6 Chapters4-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 6 of 6 · COMP5577

Privacy Attacks and Model Extraction

Outputs can disclose without a database breach

Membership inference asks whether a named record was used. Attribute inference predicts a hidden property. Model inversion reconstructs representative or sensitive features, while training-data extraction recovers memorized examples.

These claims differ in strength and must not be collapsed.

Model extraction protects a different asset

Querying a service to build a substitute targets model behavior or intellectual property.

It can coexist with privacy leakage but calls for distinct evaluation and controls such as output restriction, rate limits, watermarking and query monitoring.

Overfitting creates an observable gap

A model may produce more confident or distinctive outputs for training members than for unseen examples. Shadow models learn this difference from known member and non-member splits, then classify target outputs.

Better generalisation can reduce the signal, but evaluation still needs balanced populations and uncertainty. A high membership score is evidence about likelihood, not certainty that one named record was used.

Limit what training and interfaces reveal

Data minimisation, deduplication and privacy-aware training reduce exposure at learning time.

Output restriction, rate limits and query monitoring constrain what an observer can collect. These controls trade against utility and accountability: logs can reveal abuse while retaining sensitive prompts, and reduced confidence detail can limit legitimate diagnosis. Specify which information is necessary, who may access it and when it is deleted rather than treating privacy as a binary switch.

In this chapter

What this chapter covers

  • 01

    Membership inference and shadow models

  • 02

    Overfitting and generalization gaps

  • 03

    Attribute inference

  • 04

    White-box and black-box inversion

  • 05

    Training-data memorization and extraction

  • 06

    Model extraction and interface controls

Worked example · free

Classify a confidence-based membership attack

Q [9 marks]. An attacker trains shadow models on known member and non-member splits, then uses a target API probability vector. Name the objective, capability and controls. The marks are an AskSia study allocation, not a University marking scheme.
The staircase moves from a yes-or-no fact to increasingly specific content.The staircase moves from a yes-or-no fact to increasingly specific content. Each higher box asks outputs to reveal more about records, features or memorised training material.Outputs support increasingly specific disclosure questionsMembershipWas record used?AttributeWhat hidden trait?InversionWhat prototype?Data extractionWhich sequence?
The staircase moves from a yes-or-no fact to increasingly specific content. Each higher box asks outputs to reveal more about records, features or memorised training material.
  • 3Name the recovered fact and primary asset.
  • 3State the access and auxiliary knowledge.
  • 3Match controls to the capability.
The objective is whether a named record participated in training, so the attack is membership inference and the asset is personal training participation. Capability is black-box confidence access plus auxiliary data. Reduce overfitting and unnecessary output detail, rate-limit probing and evaluate privacy-aware training where its utility cost is acceptable.
Sia tip — Describe the recovered object before naming the attack; a prototype, exact sequence and membership bit are not interchangeable.
Glossary

Key terms

Membership Inference
An attack that estimates whether a specific record appeared in training.
Model Inversion
An attack that reconstructs representative or sensitive input features from model behavior.
Training-data Extraction
An attack that recovers exact or near-exact memorized training examples.
Generalization Gap
The performance difference between training and unseen data that can provide a membership signal.
FAQ

Privacy Attacks and Model Extraction FAQ

Does a reconstructed face prove that exact image was in training?

Not necessarily. Model inversion may recover a prototype or features that strongly activate a class without reproducing a specific record. Call it training-data extraction only when the evidence supports exact or near-exact recovery of memorized material.

Why does overfitting help membership inference?

A model may behave more confidently or distinctly on examples it fitted than on unseen examples. Shadow models learn this behavior gap from known member and non-member outputs. Better generalization can reduce the signal, although interface and privacy controls may still be needed.

How is model extraction different from training-data extraction?

Model extraction seeks the decision function or a substitute model, usually by querying outputs. Training-data extraction seeks memorized examples from the data used to train the model. The primary assets and mitigations differ even when both attacks use the same interface.

Study strategy

Exam move

Create a five-row comparison table for membership, attribute inference, inversion, training-data extraction and model extraction. Fill in the recovered object, attacker access, evaluation evidence and matching control. Revise by covering the attack names and identifying them from the recovered object alone.

Add a column for claim strength: probability of presence, inferred characteristic, prototype, exact sequence or copied behavior. Practise downgrading an overclaim, such as changing exact training-image recovery to a class-representative inversion when the evidence only shows resemblance.

Map data minimisation, deduplication, better generalisation, differential privacy, output restriction, rate limits and monitoring to the capabilities they constrain. Finally audit the privacy cost of the monitoring itself by specifying retained fields, access and deletion. A complete answer protects people without pretending accountability has no data cost.

For each row, write one observation that would support the claim and one that would be insufficient. For extraction, distinguish resemblance from an exact or near-exact sequence; for model copying, distinguish matching outputs on sampled queries from identical internals. Then choose a control and state the utility it may reduce. This evidence-control-cost triangle is a reliable structure for longer answers.

State whose privacy is at risk before selecting the defense or metric.

Working through Privacy Attacks and Model Extraction in COMP5577? Sia is AskSia’s AI Computer Science tutor — ask any COMP5577 Privacy Attacks and Model Extraction question and get a clear, step-by-step explanation grounded in how COMP5577 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 24 of your subjects at The Hong Kong Polytechnic University, plus 1,000+ more Bibles across the AskSia library.
Sia - your COMP5577 tutor, unlimited, worked the way the exam marks it
The full 4-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works