MATH7970 Chap.1 Forecast Baselines and Error Measures
Forecast Baselines and Error Measures
Benchmarks before complexity
Naive, mean and drift forecasts encode different beliefs about level and movement. A fair comparison fixes the same forecast origins and evaluates errors that were genuinely unknown when each forecast was made.
Loss determines the winner
Signed error reveals bias, MAE preserves the original unit and MSE gives large misses extra influence.
State actual minus forecast before interpreting direction, and choose the measure that fits the operational cost.
Information boundaries and benchmarks
A forecast is honest only when every input was known at its origin. Record the last observed date, target date and horizon before calculating.
Rolling-origin evaluation repeats this boundary across time, preventing a later actual value from leaking into an earlier forecast through a mean, fitted coefficient or transformation. Give every competing method identical origins and targets.
Ordinary naive challenges claims that the latest change is predictable; seasonal naive challenges models of recurring calendar structure; drift challenges models whose main advantage is a persistent slope. Several baselines reveal what extra structure a complex model actually exploits.
Error measures answer different decisions
Signed errors reveal systematic under- or over-forecasting but can cancel.
MAE reports a typical miss in the original unit and treats each additional unit equally. MSE and RMSE give large errors more influence, which fits decisions where extreme misses are disproportionately costly. Percentage errors support some cross-series comparisons yet become unstable near zero.
State the convention, unit, number of origins and operational cost before naming a winner, since changing any of those can reverse the ranking.
Average performance needs a time pattern
Plot the error sequence as well as summarising it. A low average can hide a long run of errors with one sign, deterioration after a structural break or a single exceptional miss.
Compare early and late holdout periods and identify known interventions. If one method wins only before a regime change, report that boundary.
If removing the largest error reverses the ranking, describe sensitivity rather than presenting the average as a stable property of the model.
From score to recommendation
A decision-ready conclusion names the preferred method for the tested horizon and loss, quantifies its advantage over the strongest relevant baseline and states whether that advantage matters in the response unit.
It also names the main limitation, such as a short validation window, unstable percentage denominator or break. When methods tie, retain both until another origin separates their assumptions.
In-sample fit cannot break the tie because it answers how well the methods describe known observations, not how they perform beyond the information boundary.
Stress-test the ranking
Repeat the comparison after changing one reasonable design choice: shorten the training window, move the holdout forward, replace MSE with MAE or use a seasonal benchmark where calendar repetition is visible.
The aim is not to search until a preferred method wins. It is to learn which conclusion survives defensible alternatives. If the leading method changes, identify the observation or cost assumption responsible. A robust report can still recommend one forecast, but it should explain the condition under which another method becomes preferable and preserve the original error ledger so the claim can be recomputed.
What this chapter covers
- 01
Forecast origin
- 02
Naive forecast
- 03
Mean and drift methods
- 04
Signed forecast error
- 05
MAE and MSE
- 06
Rolling-origin comparison
Worked example · free
Compare two baseline forecasts
- 3Form both forecasts from the first three observations.
- 2Compute signed errors when 26 arrives.
- 1Interpret the tie.
Key terms
- Forecast Origin
- The last date whose information is available to the forecaster.
- Naive Forecast
- A forecast carrying the latest observation forward.
- Mean Absolute Error
- The average magnitude of forecast errors without their signs.
- Mean Squared Error
- The average squared error, which emphasises larger misses.
- Bias
- A persistent directional tendency visible in signed errors.
Forecast Baselines and Error Measures FAQ
Why use a naive forecast?
It supplies a demanding simple benchmark when changes are hard to predict. A complex model that cannot beat it out of sample has not demonstrated useful additional structure. It also exposes whether the evaluation design rewards persistence rather than genuine predictive improvement.
Why can percentage error fail?
Dividing by an actual value near zero makes the percentage unstable or undefined, so scale-free appearance can conceal a poor measure. Inspect the scale of actual values before choosing it for cross-series reporting.
What does a positive error mean here?
Under actual minus forecast, a positive error means the method predicted too low and the realised value was higher than expected. Keeping that wording beside the formula prevents the common reversal during interpretation.
Exam move
Create a small error ledger with origin, forecast, actual, signed error, absolute error and squared error. Compare every new method with the same ledger so no model receives easier dates. Add a column for the assumption behind every method and another for the observation that would refute it.
Once a week, recompute naive, mean and drift forecasts from one shared origin without software, then compare the full error sequence under MAE and RMSE. Remove the largest miss and note whether the ranking changes. Finish with a recommendation naming horizon, unit, validation window and practical improvement. This routine joins arithmetic to model judgement and keeps the information boundary visible.
Revisit the ledger after a week and recompute one row from raw observations. That delayed check tests whether the notation remains understandable without the original working and exposes any forecast that quietly used future information. Archive the completed table so later model comparisons use the same verified arithmetic and definitions.
Working through Forecast Baselines and Error Measures in MATH7970? Sia is AskSia’s AI Mathematics tutor — ask any MATH7970 Forecast Baselines and Error Measures question and get a clear, step-by-step explanation grounded in how MATH7970 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
