The Hong Kong Polytechnic University · FACULTY OF COMPUTER SCIENCE

COMP5577 Chap.4 Detecting and Repairing Poisoned Models

- one subject, every graph, every model, every mark
6 Chapters3-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 4 of 6 · COMP5577

Detecting and Repairing Poisoned Models

Defense has four different jobs

Prevention controls sources, detection finds suspicious structure, robustness limits damage and recovery restores a usable model.

Activation clustering records internal representations, groups them by class, reduces dimensions, clusters and inspects suspicious groups for a trigger or label pattern.

Detection must end in reevaluation

A cluster is a hypothesis, not a verdict. After confirmed poison is removed or relabeled, retrain and repeat both clean and triggered evaluation.

The number of samples removed cannot replace attack success rate as evidence that the backdoor is gone.

Geometry needs semantic inspection

K-means and density methods can reveal separation in activation space, but a small cluster may represent a legitimate device, accent, writing style or subgroup. Treating size as maliciousness can destroy useful coverage and create unequal error.

Inspect examples, labels, candidate triggers and predictions, then test whether the suspected mechanism explains the split. The meaning comes from the threat model, not the clustering algorithm.

Recovery is a measured state

Removing confirmed poison and retraining from scratch is the clearest path, while relabeling and continued training may be faster.

Either way, preservation of clean utility and reduction of triggered attack success must be shown after repair. If a few consistent examples preserve the trigger, a high removal percentage is misleading. Monitoring then watches for recurrence, completing a loop from detection through restored trust rather than ending with a flagged cluster.

In this chapter

What this chapter covers

  • 01

    Prevention, detection, robustness and recovery

  • 02

    Outliers and anomaly assumptions

  • 03

    K-means and density-based clustering

  • 04

    Activation vectors within classes

  • 05

    Trigger inspection and confirmation

  • 06

    Repair, retraining and reevaluation

Worked example · free

Carry a suspicious cluster through repair

Q [10 marks]. An audio model shows a small activation cluster in the unlock class, and most clips share a high-frequency tone. Give the defensible workflow. The marks are an AskSia study allocation, not a University marking scheme.
The workflow returns to evaluation rather than ending at a suspicious cluster.The workflow returns to evaluation rather than ending at a suspicious cluster. Each box feeds the next, and the dashed arrow requires clean and triggered tests after repair.Activation clustering turns internal behaviour into a repair decisionActivationsGroup classReduceClusterInspectRepairretrain and reevaluate clean accuracy plus ASR
The workflow returns to evaluation rather than ending at a suspicious cluster. Each box feeds the next, and the dashed arrow requires clean and triggered tests after repair.
  • 3Preserve baseline clean and triggered metrics.
  • 3Inspect and confirm the shared mechanism.
  • 4Repair, retrain and reevaluate.
Measure baseline clean accuracy and triggered attack success, inspect the cluster for the tone and sweep the full training set for the pattern. Remove or relabel confirmed poison, preserve legitimate minority examples, retrain and repeat both metrics. Low triggered success with acceptable clean performance is recovery evidence.
Sia tip — Never delete the smaller cluster automatically; natural subgroups can be small and security geometry does not supply semantic meaning.
Glossary

Key terms

Activation Vector
The internal representation recorded for an input at a chosen model layer.
Activation Clustering
Grouping within-class activation vectors to identify samples learned through a distinct feature pattern.
Centroid
The arithmetic mean position representing a cluster in centroid-based methods.
Recovery
Removing the cause, repairing or retraining the model, and verifying restored behavior.
FAQ

Detecting and Repairing Poisoned Models FAQ

Why is a suspicious cluster not enough to delete data?

Clustering supplies geometric separation, not semantic meaning. Legitimate accents, devices, writing styles or subgroups can form a minority cluster. Inspect examples, labels, triggers and predictions before removal or a security defense may create a utility or fairness failure.

How is activation clustering ordered?

Record activations, group them by observed class, reduce dimensionality, cluster within each class, inspect suspicious groups, remove or relabel confirmed poison, retrain and reevaluate clean accuracy plus triggered attack success. The final evaluation determines whether trust was restored.

Why can removing most poison leave a backdoor?

A small number of consistent examples may be enough to preserve the trigger association. Counting removed samples measures the cleaning process, not model behavior. The repair verdict must come from triggered attack success alongside clean performance after retraining.

Study strategy

Exam move

Memorise the workflow as evidence layers rather than a list of algorithms: separation, confirmation and recovery. For every threshold, state the false-positive and false-negative consequence. Practise cases where the smaller cluster is legitimate so that cluster size never becomes an automatic security label.

Write the activation-clustering steps from memory, then justify why grouping by observed class occurs before dimensionality reduction and clustering. Compare K-means assumptions with density-based noise detection and ask what a large poison population does to each. For repair, list the preserved baseline metrics, the semantic inspection, the cleaning action, retraining and final clean plus triggered tests.

Use a separate line for fairness or utility damage caused by false removal. The goal is a chain of evidence, not a favourite clustering algorithm. Create a decision log for three suspicious clusters: one poisoned, one rare but legitimate and one inconclusive. Record the representation used, the threshold, the semantic evidence and the cost of removal. Then swap the cluster sizes without changing their meanings.

Your decisions should follow the evidence, not the minority label. This drill makes false-positive harm concrete and prevents detection output from being treated as an automatic repair command. Preserve a pre-cleaning baseline so recovery claims have an honest comparison point.

Working through Detecting and Repairing Poisoned Models in COMP5577? Sia is AskSia’s AI Computer Science tutor — ask any COMP5577 Detecting and Repairing Poisoned Models question and get a clear, step-by-step explanation grounded in how COMP5577 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 24 of your subjects at The Hong Kong Polytechnic University, plus 1,000+ more Bibles across the AskSia library.
Sia - your COMP5577 tutor, unlimited, worked the way the exam marks it
The full 3-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works