ECON4916 Chap.6 Selecting and Validating Forecast Models
Selecting and Validating Forecast Models
Model selection asks whether extra flexibility captures stable signal or merely follows noise in the estimation sample. This chapter decomposes forecast error into innovation uncertainty and parameter-estimation error, then compares in-sample penalties with direct out-of-sample evaluation.
Penalised residual variance, information criteria and related measures reward fit while charging for complexity, but their directions and penalties differ. Rolling and recursive forecast designs preserve time order and answer different questions about stability. Ridge and lasso extend the selection problem by shrinking coefficients, with lasso able to set some estimates exactly to zero.
Throughout, candidate models must use aligned samples, forecast horizons and loss functions. The closing clinic links the selected criterion to the actual financial decision and treats a narrow performance gap as uncertain evidence rather than a permanent leaderboard.
What this chapter covers
- 01
Fit and complexity trade-off
- 02
Penalised in-sample criteria
- 03
Time-ordered holdout evaluation
- 04
Rolling and recursive windows
- 05
Ridge and lasso shrinkage
- 06
Leakage and fair model comparison
Worked example · free
Penalised variance across candidate models
- +1For Model A, calculate 150 divided by 118 = 1.2712.
- +1For Model B, calculate 138 divided by 115 = 1.2000.
- +1For Model C, calculate 132 divided by 111 = 1.1892.
- +1Select Model C because the stated penalised residual variance is minimised.
- +1Qualify the result as an in-sample criterion ranking and note that Model B is close.
Key terms
- Model Complexity
- The flexibility added by estimated parameters, lag orders, predictors or other adjustable structure.
- Information Criterion
- A model-ranking measure combining goodness of fit with a penalty for estimated complexity.
- Holdout Sample
- Time-ordered observations reserved from fitting and used to evaluate forecasts after estimation.
- Recursive Window
- An estimation sample that expands as each new forecast origin is reached.
- Rolling Window
- A fixed-length estimation sample that advances by dropping old observations and adding new ones.
- Data Leakage
- Use of future or evaluation-period information while fitting, transforming or selecting a forecast model.
Selecting and Validating Forecast Models FAQ
Why does training error favour larger models?
Additional regressors or lags let a fitted model reproduce more features of the estimation sample. In ordinary least squares, residual sum of squares cannot rise when regressors are added. Without a complexity penalty or later evaluation, training fit rewards flexibility even when it captures noise that will not persist.
How should I split time-series data?
Preserve time order. Estimate on observations available before a forecast origin and evaluate on later outcomes. A random split can train on future information and create leakage. Choose rolling or recursive windows based on whether older data remain informative, then apply the same rule to every candidate.
What is the difference between ridge and lasso?
Ridge penalises squared coefficient magnitudes and usually shrinks all predictors without setting them exactly to zero. Lasso penalises absolute magnitudes and can create a sparse coefficient vector. Both require a tuning parameter chosen with a valid time-respecting evaluation design and generally need predictor scaling.
Can information criteria and holdout loss disagree?
Yes. Information criteria estimate a fit-complexity trade-off inside the estimation sample, while holdout loss measures realised forecasting performance on later observations. Penalty choices, sample variability and structural change can produce different rankings. Explain what each comparison targets instead of treating disagreement as an arithmetic error.
Exam move
Practise model selection with deliberately small tables so every comparison can be audited. For each candidate, record the common estimation window, parameter count, fit measure and criterion direction. Recompute one penalised criterion by hand and explain why its penalty rises with complexity.
Next, design a pseudo-out-of-sample exercise on a timeline: mark the training window, forecast origin, target date and evaluation loss. Repeat it under rolling and recursive windows and state which stability assumption each design favours.
When using ridge or lasso, standardise predictors using training information only, choose the tuning parameter on time-respecting validation folds and apply the stored transformation to later data. Maintain a leakage checklist covering scaling, feature construction, hyperparameter choice and model ranking. Report the number of forecast errors and inspect whether one extreme period drives the winner.
If rankings are close, say so and compare residual structure or performance in decision-relevant states rather than claiming a permanent best model. End every answer by linking the loss function and horizon to the intended financial use, because a criterion is meaningful only relative to the decision it serves.
Working through Selecting and Validating Forecast Models in ECON4916? Sia is AskSia’s AI Economics tutor — ask any ECON4916 Selecting and Validating Forecast Models question and get a clear, step-by-step explanation grounded in how ECON4916 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.