MATH7970 Chap.9 Forecasting and Model Comparison
Forecasting and Model Comparison
Conditional means and widening uncertainty
Point forecasts replace future innovations by their conditional mean, while intervals include their possible effects. Persistence carries uncertainty forward across horizons.
Common evaluation conditions
Models must face identical training information, origins, horizons, targets and loss measures.
Rolling-origin evaluation reveals whether one apparent advantage depends on a lucky holdout.
Construct the conditional forecast
Known observations and recoverable past innovations enter the forecast recursion. Future innovations are replaced by their conditional mean of zero for the point path. At longer horizons, earlier forecasts replace unavailable observations.
Stable AR contributions shrink, finite MA contributions disappear and integrated forecasts accumulate predicted changes. Label every term by date so a value from the wrong horizon cannot enter unnoticed.
Carry uncertainty through the dynamics
Each future date introduces a new unknown innovation. Model coefficients determine how those shocks also affect later horizons, usually widening intervals.
An interval must name its horizon, probability level and scale. For transformed models, return both point and uncertainty statements to the intended reporting scale rather than attaching difference-scale limits directly to levels.
Update states before refitting parameters
When the next actual arrives, its forecast error becomes new information and changes subsequent forecasts.
This state update occurs even if coefficients remain fixed. Re-estimation is separate and needs enough evidence to justify changing parameters. Distinguishing the two prevents every surprise from being treated as proof that the model itself changed.
Compare forecasts under common conditions
Fix training windows, origins, horizons, targets and loss measures before inspecting results.
Report performance by horizon when the operational decision cares about a particular distance. Evaluate interval coverage and width together: narrow intervals that miss are overconfident, while wide intervals can cover without being useful.
Prefer gains that persist across origins and exceed a relevant naive or seasonal-naive benchmark by an operationally meaningful amount.
Use baselines that share the horizon
A one-step naive forecast is not automatically the correct comparator for a twelve-step seasonal forecast. Build each baseline at the same origin and horizon as the candidate, using only information then available.
For direct multi-step models, compare the target forecast itself rather than intermediate fitted values. If a complex model wins only at horizons the decision never uses, the statistical advantage has little practical value. Present a horizon-by-horizon table and name the range over which the ranking is stable.
What this chapter covers
- 01
minimum-MSE forecast
- 02
multi-step recursion
- 03
forecast updating
- 04
forecast intervals
- 05
rolling origins
- 06
out-of-sample loss
Worked example · free
Update an AR forecast
- 3Compute the first forecast.
- 2Find its error.
- 2Update the next forecast.
Key terms
- Point Forecast
- A single conditional prediction, often the conditional mean.
- Forecast Interval
- A range reflecting uncertainty from future innovations and estimation.
- Rolling Origin
- Repeated evaluation after advancing the information date.
- Holdout Sample
- Observations excluded from fitting and reserved for evaluation.
- Forecast Updating
- Revising predictions after a new observation enters the information set.
Forecasting and Model Comparison FAQ
Why do intervals widen with horizon?
Each additional future date introduces another unknown innovation, and persistent dynamics carry earlier uncertainty forward. The expanding interval is therefore a model implication rather than a decorative addition.
Why not compare fitted residuals only?
They use observations that influenced estimation and reward flexibility, so they do not reproduce the future forecasting task. Held-out origins reveal performance after the information boundary is respected.
Which loss measure is best?
The choice depends on decision cost and scale. MAE is interpretable, RMSE emphasises large misses and percentages can fail near zero. Report a scale-dependent measure whenever operational miss size matters to the decision.
Exam move
Create rolling-origin tables by hand for a short series. Keep each model's origin and horizon aligned, then explain why the chosen loss represents the actual forecasting decision. Build a rolling-origin worksheet with separate rows for short, seasonal and longer horizons. Force all candidates to use identical information and targets, then report MAE and RMSE by horizon.
Calculate interval coverage and average width on the same holdout. Update one forecast after every realised value without re-estimating, then contrast that state update with a deliberately refitted model. Finish by naming the operational horizon and explaining whether measured improvement justifies extra complexity. Preserve point loss and interval performance together.
A model improving one while damaging the other should be judged against the actual decision rather than receiving an automatic overall win. Label the horizon on every graph, table and interval before comparing forecast quality.
Working through Forecasting and Model Comparison in MATH7970? Sia is AskSia’s AI Mathematics tutor — ask any MATH7970 Forecasting and Model Comparison question and get a clear, step-by-step explanation grounded in how MATH7970 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
