The University of Hong Kong · FACULTY OF STATISTICS

STAT8003 Chap.8 Model Checking and Forecast Comparison

- one subject, every graph, every model, every mark
7 Chapters4-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 8 of 8 · STAT8003

Model Checking and Forecast Comparison

Build an auditable modelling chain

This chapter follows the published project workflow from raw record to held-out comparison. Start with the time plot and a stationarity decision, justify any transformation, and use sample ACF and PACF evidence to propose several candidate orders.

Each transition needs a sentence explaining what evidence licensed it.

Separate adequacy from preference

After estimation, use Ljung-Box tests at several lag choices and an over-parameterisation check to challenge the fitted dynamics. A lower information criterion does not rescue residual dependence.

Compare AIC or another stated criterion only among candidates whose residual evidence is defensible, and record how estimates respond to plausible extra terms.

Keep the holdout outside construction

The final observations are removed before fitting and remain unseen through model selection. They are revealed only for forecast comparison after one specification has been chosen. This protects evaluation from tuning.

Report horizon, scale and error behaviour, then connect unusual misses to assumptions, transformations or structure that the model did not represent.

In this chapter

What this chapter covers

  • 01

    Time plot and transformation

  • 02

    ACF and PACF candidates

  • 03

    Parameter estimation

  • 04

    Ljung-Box diagnostics

  • 05

    Over-parameterisation check

  • 06

    AIC selection

  • 07

    Held-out forecast comparison

Worked example · free

Choose adequacy before AIC

Q [8 marks]. AskSia-authored practice. The marks shown are an AskSia study allocation, not a University marking scheme. Candidate A has lower AIC but significant residual autocorrelation. Candidate B has slightly higher AIC and acceptable residual checks. Which proceeds to holdout forecasting?
Model selection is a funnel rather than a race to the lowest AIC.Model selection is a funnel rather than a race to the lowest AIC. Candidates must pass stationarity and residual checks before held-out forecast comparison.stationary candidate modelsadequate residuals at several Kmodel selectionholdout forecastforecast only after diagnosis
Model selection is a funnel rather than a race to the lowest AIC. Candidates must pass stationarity and residual checks before held-out forecast comparison.
  • 3Identify the adequacy failure.
  • 3Restrict selection to defensible candidates.
  • 2State the holdout's role.
Candidate B proceeds because residual adequacy comes before information-criterion preference. Candidate A still leaves systematic dependence unexplained. After selection, B generates forecasts for the values reserved before fitting, and those predictions are compared with the observed holdout.
Sia tip — Write the words 'among adequate models' immediately before any AIC conclusion.
Glossary

Key terms

Model Adequacy
The extent to which a fitted model removes systematic dependence targeted by its diagnostics. Adequacy is assessed before information criteria are used to choose among defensible candidates.
Information Criterion
A relative model score balancing fit against complexity, used among defensible candidates. A smaller value ranks models but does not prove that their residuals are free of dependence.
Holdout Forecast
A prediction evaluated against observations excluded from model fitting and selection. Keeping those values hidden protects the comparison from tuning and separates construction evidence from evaluation evidence.
FAQ

Model Checking and Forecast Comparison FAQ

Why should residual tests be run at several lags?

A single lag choice examines only one aggregation of residual autocorrelations. The project specifically asks for several K values so adequacy is not made to depend on one convenient window of dependence.

What is the purpose of over-parameterising a fitted model?

Adding a plausible extra term tests whether established estimates remain stable and whether omitted dependence may be distorting them. The goal is diagnostic comparison, not automatic preference for the larger specification.

When should the holdout values be inspected?

Only after transformation, candidate specification, estimation, residual checking and model selection are complete. Earlier inspection leaks evaluation information into construction and makes the final forecast comparison less independent.

What makes a model-selection argument auditable?

Define the training and holdout periods, explain every transformation, record how ACF and PACF evidence generated candidates, and report residual tests at stated lags. Restrict information-criterion comparison to models that survive adequacy checks. Preserve the selected specification before opening the holdout, then report forecast horizon, error scale and important misses.

Each table or plot should be followed by a sentence naming the decision it supports.

Why keep model-selection and forecast tables separate?

Selection tables use training evidence to choose a specification, while forecast tables evaluate that fixed choice on held-out observations. Combining them can hide whether future values influenced tuning. Separate tables make the evaluation boundary visible and auditable.

Study strategy

Exam move

Memorise the modelling workflow as a chain of permissions. A plot permits a transformation decision; stationarity permits correlation-based specification; estimation permits residual diagnosis; adequate residuals permit information-criterion comparison; selection permits held-out forecasting. In written work, include a sentence at every transition explaining what evidence licensed the next step.

Start practice with a blank flowchart and fill it from memory: define the training and holdout blocks, inspect the time plot, justify transformation, identify candidates from sample ACF and PACF, estimate, check residuals at several K values, try an over-parameterised alternative, compare adequate models, review normality and forecast the withheld observations.

For every arrow, write the evidence that allows movement to the next box. Create deliberate conflict cases in which the lowest AIC model has residual autocorrelation, or a more complex model changes earlier estimates sharply. Explain why adequacy and stability matter before preference.

Keep the holdout hidden during all tuning; if it has influenced order choice, label the evaluation compromised and propose a fresh comparison design. When forecasts are finally evaluated, state the horizon, scale and error measure, then inspect individual misses as well as averages. Connect forecast errors back to transformations, seasonal structure, changing variance or shocks not represented in the model.

In the report, use headings to expose this chain rather than to repeat output. A figure or table should be followed by a decision sentence explaining what changed because of it. The final conclusion should distinguish what the fitted model captured, what the diagnostics could not reject and what remains a limitation.

Working through Model Checking and Forecast Comparison in STAT8003? Sia is AskSia’s AI Statistics tutor — ask any STAT8003 Model Checking and Forecast Comparison question and get a clear, step-by-step explanation grounded in how STAT8003 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 52 of your subjects at The University of Hong Kong, plus 1,000+ more Bibles across the AskSia library.
Sia - your STAT8003 tutor, unlimited, worked the way the exam marks it
The full 4-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works