BPS3082 Chap.7 Model Fitting and Diagnostics
Model Fitting and Diagnostics
Model fitting is an iterative test of adequacy, not a search for one winning number. Optimisation estimates parameters under a structural model and an error model, while diagnostics ask whether the remaining differences resemble noise or retain systematic information.
An overlay can reveal missed phases; residuals against time or predictions can show curvature, runs of one sign or a funnel; parameter estimates can expose impossible signs, poor precision or strong correlation. Each view answers a different question. Residual sum of squares compresses deviations into a useful objective but hides their location and direction.
Weighting changes which residuals dominate, so it should respond to the observed variance pattern rather than serve as an unexplained software setting. A time trend often points to missing structural behaviour, while a funnel can indicate non-constant residual variance.
The repair should match the pattern: a variance-model problem is not automatically fixed by another compartment, and a missing phase is not solved by cosmetic weighting. AIC balances likelihood fit against parameter count and is meaningful as a relative comparison among candidates fitted to the same data on the same likelihood basis. Lower AIC is evidence, not a biological verdict.
Candidate selection combines overlays, residuals, AIC, parameter credibility and performance for the intended simulation. When two models perform similarly, the simpler identifiable model is preferred unless added structure has a necessary mechanistic or predictive role. A reproducible selection record preserves the dataset version, candidate definition, error model, uncertainty and reasons for rejecting alternatives.
Repeating the same checks after every revision prevents improvement in one metric from hiding deterioration elsewhere.
What this chapter covers
- 01
Predicted-versus-observed plots
- 02
Residual patterns
- 03
Residual sum of squares
- 04
Weighting models
- 05
Akaike Information Criterion
- 06
Parsimony and prediction
- 07
Structural bias versus variance mismatch
- 08
Like-for-like candidate comparison
- 09
Parameter stability after model revision
- 10
Use-specific predictive performance
Worked example · free
Reject a misleading numerical winner
- +2Recognise the ordered residual signs as systematic time-dependent bias.
- +2Do not select Model B from its lower AIC alone because the structure is visibly inadequate.
- +2Investigate a mechanistic revision and compare all diagnostics after refitting.
Key terms
- Residual
- The difference between an observed value and its model prediction.
- Residual Pattern
- Systematic structure in errors that signals model or variance misspecification.
- AIC
- A relative criterion balancing model fit against parameter count for like-for-like candidates.
- Weighting
- The rule controlling how residuals at different scales contribute to fitting.
- Parsimony
- Preference for the simplest model that adequately explains data and supports its purpose.
Model Fitting and Diagnostics FAQ
Is the model with the lowest AIC always best?
No. AIC is a relative comparison on the same data and likelihood basis. Residual patterns, parameter stability, mechanism and intended prediction also matter, especially when AIC differences are small. A candidate with slightly lower AIC but ordered residual bias is not automatically more adequate; investigate the missed structure and repeat the full comparison.
What does a funnel-shaped residual plot suggest?
It suggests residual variance changes with prediction magnitude. Investigate the error model, response scale or weighting before adding a structural PK or PD component that does not address the variance pattern. Refit and verify that the funnel decreases without creating time trends, unstable parameters or a new failure near the decision range.
Why prefer fewer parameters when fit is equal?
Extra parameters consume information, can be unstable and complicate prediction. If diagnostics and intended performance are equivalent, the simpler model is easier to identify, explain and apply. Parsimony does not mean ignoring necessary biology; it means requiring every added parameter to have evidence and a useful role.
Assessment move
Use the same diagnostic sequence for every candidate: overlay observations and predictions, inspect residuals against time and predictions, compare AIC on the same basis, audit parameter signs, precision and correlation, then test the intended simulation. For each residual pattern, write the likely failure class, the specific repair and the diagnostic that should improve if the repair is correct.
Contrast time trends with funnels so structural and variance problems are not treated interchangeably. Keep a candidate-selection table that records dataset, weighting, parameter count, strengths, rejection reasons and limitations rather than only the winner. Practise cases where the lowest AIC conflicts with residual evidence and explain why the visual pattern matters.
After any change, repeat the full sequence to check for displaced problems. Include data-provenance and unit checks before excluding surprising observations or adding complexity. Finish every selection answer with the preferred model, two supporting diagnostics, its intended use and the most important unresolved limitation. Create four residual sketches: random scatter, a time trend, curvature and a funnel.
For each, identify the likely failure class without overdiagnosing a unique cause, propose one focused revision and state what the new diagnostic should show. Repeat the exercise after changing the response scale or weighting so the objective function remains explicit. Maintain a compact audit trail for any excluded point, distinguishing transcription evidence from poor fit.
Before accepting a candidate, test predictions specifically near the concentration, effect or regimen boundary that matters for the development decision rather than relying on average performance.
