STAT8003 Chap.8 Model Checking and Forecast Comparison
Model Checking and Forecast Comparison
Build an auditable modelling chain
This chapter follows the published project workflow from raw record to held-out comparison. Start with the time plot and a stationarity decision, justify any transformation, and use sample ACF and PACF evidence to propose several candidate orders.
Each transition needs a sentence explaining what evidence licensed it.
Separate adequacy from preference
After estimation, use Ljung-Box tests at several lag choices and an over-parameterisation check to challenge the fitted dynamics. A lower information criterion does not rescue residual dependence.
Compare AIC or another stated criterion only among candidates whose residual evidence is defensible, and record how estimates respond to plausible extra terms.
Keep the holdout outside construction
The final observations are removed before fitting and remain unseen through model selection. They are revealed only for forecast comparison after one specification has been chosen. This protects evaluation from tuning.
Report horizon, scale and error behaviour, then connect unusual misses to assumptions, transformations or structure that the model did not represent.
What this chapter covers
- 01
Time plot and transformation
- 02
ACF and PACF candidates
- 03
Parameter estimation
- 04
Ljung-Box diagnostics
- 05
Over-parameterisation check
- 06
AIC selection
- 07
Held-out forecast comparison
Worked example · free
Choose adequacy before AIC
- 3Identify the adequacy failure.
- 3Restrict selection to defensible candidates.
- 2State the holdout's role.
Key terms
- Model Adequacy
- The extent to which a fitted model removes systematic dependence targeted by its diagnostics. Adequacy is assessed before information criteria are used to choose among defensible candidates.
- Information Criterion
- A relative model score balancing fit against complexity, used among defensible candidates. A smaller value ranks models but does not prove that their residuals are free of dependence.
- Holdout Forecast
- A prediction evaluated against observations excluded from model fitting and selection. Keeping those values hidden protects the comparison from tuning and separates construction evidence from evaluation evidence.
Model Checking and Forecast Comparison FAQ
Why should residual tests be run at several lags?
A single lag choice examines only one aggregation of residual autocorrelations. The project specifically asks for several K values so adequacy is not made to depend on one convenient window of dependence.
What is the purpose of over-parameterising a fitted model?
Adding a plausible extra term tests whether established estimates remain stable and whether omitted dependence may be distorting them. The goal is diagnostic comparison, not automatic preference for the larger specification.
When should the holdout values be inspected?
Only after transformation, candidate specification, estimation, residual checking and model selection are complete. Earlier inspection leaks evaluation information into construction and makes the final forecast comparison less independent.
What makes a model-selection argument auditable?
Define the training and holdout periods, explain every transformation, record how ACF and PACF evidence generated candidates, and report residual tests at stated lags. Restrict information-criterion comparison to models that survive adequacy checks. Preserve the selected specification before opening the holdout, then report forecast horizon, error scale and important misses.
Each table or plot should be followed by a sentence naming the decision it supports.
Why keep model-selection and forecast tables separate?
Selection tables use training evidence to choose a specification, while forecast tables evaluate that fixed choice on held-out observations. Combining them can hide whether future values influenced tuning. Separate tables make the evaluation boundary visible and auditable.
Exam move
Memorise the modelling workflow as a chain of permissions. A plot permits a transformation decision; stationarity permits correlation-based specification; estimation permits residual diagnosis; adequate residuals permit information-criterion comparison; selection permits held-out forecasting. In written work, include a sentence at every transition explaining what evidence licensed the next step.
Start practice with a blank flowchart and fill it from memory: define the training and holdout blocks, inspect the time plot, justify transformation, identify candidates from sample ACF and PACF, estimate, check residuals at several K values, try an over-parameterised alternative, compare adequate models, review normality and forecast the withheld observations.
For every arrow, write the evidence that allows movement to the next box. Create deliberate conflict cases in which the lowest AIC model has residual autocorrelation, or a more complex model changes earlier estimates sharply. Explain why adequacy and stability matter before preference.
Keep the holdout hidden during all tuning; if it has influenced order choice, label the evaluation compromised and propose a fresh comparison design. When forecasts are finally evaluated, state the horizon, scale and error measure, then inspect individual misses as well as averages. Connect forecast errors back to transformations, seasonal structure, changing variance or shocks not represented in the model.
In the report, use headings to expose this chain rather than to repeat output. A figure or table should be followed by a decision sentence explaining what changed because of it. The final conclusion should distinguish what the fitted model captured, what the diagnostics could not reject and what remains a limitation.
Working through Model Checking and Forecast Comparison in STAT8003? Sia is AskSia’s AI Statistics tutor — ask any STAT8003 Model Checking and Forecast Comparison question and get a clear, step-by-step explanation grounded in how STAT8003 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.