ELEC5618 · Software Quality Engineering
Quality Engineering for ML/AI Systems
Week 10 of University of Sydney ELEC5618 Software Quality Engineering is a guest lecture applying software-quality thinking to machine-learning and AI systems: why probabilistic model behaviour breaks classical testing assumptions, the eval pipeline and launch gates, what you can and cannot unit-test in an ML system, and the three-layer quality stack that adds ML gates on top of standard software engineering. It frames the practical case study the targeted project draws on.
What this chapter covers
- 01Deterministic (same input → same output) vs probabilistic software (same input → a distribution of outputs); quality becomes multi-dimensional and conflicting
- 02Model quality dimensions: correctness/accuracy, robustness, fairness, latency/efficiency, safety, consistency; 'metrics are decisions'
- 03The eval pipeline: model → offline eval dataset → metrics → online A/B experiment → launch decision → continuous monitoring
- 04Offline eval-data concerns: source (labelled/synthetic/logs), train-eval contamination (leakage), statistical power, refresh cadence
- 05Online experiments: top-line metrics, guardrail metrics, slice-based metrics (by language/topic/user), resource cost; human eval when metrics can't capture a dimension
- 06What you CAN unit-test: the system around the model (input validation, output formatting, safety filters, API contracts, latency SLOs) — not the core inference
- 07Traditional testing culture: testing pyramid (unit → integration → end-to-end); fast, hermetic, readable tests; code review; CI/CD presubmit
- 08Three-layer quality stack: L1 code quality, L2 ML/AI quality, L3 process quality — you cannot skip layers
Designing quality checks for a spam classifier
- +1(a) Two dimensions: accuracy (correctly flagging spam) and safety/robustness (not wrongly blocking legitimate mail). They conflict because tightening the filter to catch more spam (higher recall) tends to block more legitimate messages (more false positives) — 'metrics are decisions', so you must choose the trade-off explicitly.
- +1(b) You CAN unit-test the system around the model — input validation/preprocessing, output formatting, the safety filter and the API contract/latency SLO. You cannot meaningfully unit-test the model's core inference itself, because its output is probabilistic; that is evaluated statistically, not asserted.
- +1(c) Map to the stack: the unit test → Layer 1 (code quality); the offline accuracy eval → Layer 2 (ML/AI quality — eval datasets and metrics); the design-doc review → Layer 3 (process quality).
- +1State the principle: you cannot skip layers — ML/AI adds gates on top of the standard software-engineering process, and because models will sometimes fail, you manage with monitoring plus fast rollback rather than trying to be perfect before launch.
Key terms
- Deterministic vs probabilistic software
- Traditional software is deterministic (the same input always yields the same output); an ML/AI system is probabilistic (the same input yields a distribution of possible outputs), which makes quality multi-dimensional and its dimensions often conflicting.
- Model quality dimensions
- Correctness/accuracy, robustness (graceful degradation on odd inputs), fairness (equal performance across subgroups), latency/efficiency, safety (refusing harmful requests) and consistency (similar inputs give similar outputs). Which metric you choose defines what 'quality' means.
- Eval pipeline
- The ML quality flow: model → offline eval dataset → metrics computation → online A/B experiment → launch decision → continuous monitoring. Each stage is a quality-engineering decision.
- What you can unit-test
- In an ML system you unit-test the deterministic system around the model — input validation and preprocessing, output formatting, safety filters and guardrails, and integration points like API contracts and latency SLOs — not the model's probabilistic core inference.
- Three-layer quality stack
- Layer 1 code quality (unit/integration tests, review, CI/CD), Layer 2 ML/AI quality (eval datasets, metrics, slice eval, regression, human eval) and Layer 3 process quality (design docs, experiment/launch reviews, monitoring). ML/AI adds gates on top of standard engineering — you cannot skip layers.
- Living with imperfection
- The recognition that ML systems will sometimes fail, so quality is managed with continuous monitoring and fast rollback and an eval → launch → monitor → improve loop, rather than by trying to be perfect before launch. Catching mistakes at design is far cheaper than after launch.
Quality Engineering for ML/AI Systems FAQ
Why does classical testing not fully work for ML/AI systems?
Because the model's behaviour is probabilistic — the same input can produce different outputs — so you cannot simply assert an expected output the way you would for deterministic code. Quality also becomes multi-dimensional and the dimensions conflict (safety vs helpfulness, accuracy vs latency). You still test the deterministic system around the model, but the model itself is evaluated statistically with datasets, online experiments and human eval.
What can and can't you unit-test in an ML system?
You can unit-test everything around the model: input validation and preprocessing, output formatting and postprocessing, safety filters and guardrails, and integration points such as API contracts and latency SLOs. You cannot meaningfully unit-test the model's core inference, because its output is probabilistic; that is what the offline eval datasets, online A/B experiments and human evaluation are for.
What is the three-layer quality stack?
Layer 1 is code quality (unit and integration tests, code review, readability, CI/CD); Layer 2 is ML/AI quality (eval datasets, metrics, slice-based evaluation, regression testing, human eval); Layer 3 is process quality (design-doc, experiment, eval and launch reviews, plus a monitoring plan). The point is that ML/AI adds gates on top of the ordinary software process, and you cannot skip a layer.
Can AI help me with the ML/AI quality material?
Yes. Sia can walk through the eval pipeline, help you reason about which quality dimensions conflict, and sort activities into the three-layer stack. Use it to understand the concepts; it does not do graded assessment, and the University of Sydney academic-integrity policy applies.
Exam move
This guest-lecture week is conceptual, so learn it as a set of clean distinctions and lists. Be able to explain deterministic versus probabilistic software in one sentence and why it makes quality multi-dimensional. Memorise the eval pipeline as an ordered flow (model → offline eval → metrics → online A/B → launch → monitor) and the three-layer quality stack (code, ML, process) with an example activity in each — sorting activities into layers is a natural exam task. Hold onto two ideas: you unit-test the system around the model but evaluate the model statistically, and 'metrics are decisions' so you must choose your quality trade-offs explicitly. This chapter frames the targeted project, so connect it to your case-study work. Confirm the exam format on Canvas.
Working through Quality Engineering for ML/AI Systems in ELEC5618? Sia is AskSia’s AI Software Engineering tutor — ask any ELEC5618 Quality Engineering for ML/AI Systems question and get a clear, step-by-step explanation grounded in how ELEC5618 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
