ETF5952 · Quantitative Methods for Risk Analysis
Model Risk and VaR Backtesting
This topic sits near the end of ETF5952 Quantitative Methods for Risk Analysis at Monash University, where the unit turns from building risk models to trusting them. It has two halves: naming the two sources of model risk (misspecification versus estimation) and choosing a model in-sample with AIC/BIC, then backtesting a VaR model out-of-sample by building the hit sequence of violations and running the unconditional (Kupiec), independence and conditional coverage tests. It closes on the Basel market-risk framing, where a 2.5% Expected Shortfall sets capital while VaR violations are used for backtesting.
What this chapter covers
- 011. Model risk - the risk the model itself misleads; it splits into misspecification and estimation risk
- 022. Misspecification risk - the wrong structure (e.g. a Normal model for fat-tailed returns); a bias more data cannot cure
- 033. Estimation risk - the right structure but imprecise parameters from a finite sample; shrinks as T grows
- 044. AIC and BIC - penalised-likelihood criteria; choose the SMALLEST; BIC penalises complexity more so favours simpler models
- 055. Estimation windows - fixed, recursive (expanding) and rolling schemes for re-estimating parameters out-of-sample
- 066. Hit sequence - the 0/1 record where a hit is a return worse than the forecast VaR; expected hits = T₀·p
- 077. Unconditional coverage (Kupiec) - a likelihood-ratio test that the violation RATE equals p, distributed χ² with 1 df
- 088. Independence and conditional coverage - do violations cluster in time, and do the rate AND timing both hold (χ² with 2 df)
- 099. Reading a rejection - too many hits means risk is understated; too few means overstated; clustered means poor timing
- 1010. Basel Expected Shortfall - 2.5% ES is the capital measure, with 1% and 2.5% VaR retained for backtesting against the prior 12 months
A Kupiec unconditional coverage backtest of a 5% one-day VaR
- +1Expected violations and observed rate. At coverage p = 0.05 over T₀ = 500 days the expected number of violations is T₀·p = 500(0.05) = 25. The observed count is x = 38, so the observed violation rate is π̂ = x/T₀ = 38/500 = 0.076, i.e. 7.6% - clearly above the claimed 5%.
- +1State the hypotheses. H₀: Pr(Hit_t = 1) = 0.05 (the model has correct unconditional coverage) versus H_a: Pr(Hit_t = 1) ≠ 0.05. This is a two-sided test of the average violation rate.
- +1Write the Kupiec statistic. With T₀ - x = 462, LR_uc = -2[(T₀ - x)ln(1 - p) + x·ln p - (T₀ - x)ln(1 - π̂) - x·ln π̂], comparing the likelihood of the hits under the claimed rate p with the likelihood under the observed rate π̂.
- +1Substitute the numbers. LR_uc = -2[462·ln(0.95) + 38·ln(0.05) - 462·ln(0.924) - 38·ln(0.076)] = -2[-23.698 - 113.838 + 36.518 + 97.927].
- +1Evaluate. The bracket sums to -3.091, so LR_uc = -2(-3.091) = 6.18. The statistic is non-negative, as a likelihood ratio must be, and would be 0 only if π̂ equalled p exactly.
- +1Decide and interpret. Compare with the chi-square (1 df) critical value: 6.18 > 3.841, so we reject H₀ at the 5% level - the model fails the unconditional coverage test. Because π̂ = 7.6% exceeds p = 5% (too many violations), the model underestimates risk: its VaR is too small, so capital based on it would be set too low.
Key terms
- Model risk
- The risk of loss or of bad decisions caused by using a model that does not match reality. It has two sources: misspecification risk (the wrong structure) and estimation risk (imprecise parameters). Managing it means choosing the model carefully in-sample and validating it out-of-sample by backtesting.
- Misspecification vs estimation risk
- Misspecification risk is choosing the wrong model form - for example a Normal-innovation model for returns that actually have heavy tails or a leverage effect - producing a bias that more data cannot remove. Estimation risk is having the right form but parameters that are imprecise because they are estimated from a finite sample; this error shrinks as the sample size T grows. A simpler model has less estimation risk but more misspecification risk, and vice versa.
- AIC and BIC
- Information criteria that select a model by trading off fit against complexity: AIC = (-2 ln L + 2K)/T and BIC = (-2 ln L + K ln T)/T, where L is the maximised likelihood, K the number of parameters and T the sample size. You choose the model with the smallest value. Because ln T exceeds 2 for any realistic sample, BIC penalises extra parameters more heavily and tends to select more parsimonious models.
- Estimation window
- The rule for choosing which data estimate the model as a backtest is rolled forward. A fixed window estimates parameters once and holds them; a recursive (expanding) window re-estimates on all data seen so far, so it grows; a rolling window re-estimates on a fixed-length block of the most recent observations, dropping the oldest. Longer windows reduce estimation noise but adapt slowly; shorter windows adapt fast but are noisier.
- Hit sequence (violation)
- The 0/1 record a VaR model generates out-of-sample. A hit (violation) on day t is Hit_t(p) = 1 if the return r_t falls below the negative VaR, i.e. the loss exceeds the forecast VaR, and 0 otherwise. The total hits x and the violation rate π̂ = x/T₀ are the inputs to the coverage tests; a correctly calibrated model has an expected T₀·p hits.
- Unconditional coverage test (Kupiec)
- A test of whether the average violation rate equals the promised p. The Kupiec likelihood-ratio statistic LR_uc compares the likelihood of the observed hits under the claimed rate p with that under the observed rate π̂, and is distributed chi-square with 1 degree of freedom under H₀. A large statistic (beyond 3.841 at the 5% level) rejects: the VaR is mis-calibrated.
- Independence and conditional coverage tests
- The independence test checks whether violations cluster in time, typically with a first-order Markov comparison of the probability of a hit following a hit versus following a non-hit; clustering means the model is not adapting to changing risk. The conditional coverage test combines the correct rate and independence into one hypothesis (Pr(Hit = 1 | past information) = p), distributed chi-square with 2 degrees of freedom - the most comprehensive of the three.
- Basel Expected Shortfall
- Under the current Basel market-risk framework the capital measure is a 2.5% Expected Shortfall (the average loss beyond the 2.5% VaR, computed daily bank-wide and per desk), because ES accounts for the size of tail losses, not just their frequency. VaR at 1% and 2.5% (one-day) is retained and backtested against the prior 12 months of returns. Note ES(p) is always at least as large as VaR(p).
Model Risk and VaR Backtesting FAQ
What is the difference between misspecification risk and estimation risk, and which one does more data fix?
Misspecification risk means you chose the wrong model form - for instance a Normal model for returns that really have fat tails or a leverage effect - so the model is biased in a way that persists no matter how much data you have; more data does not fix it, only changing the model does. Estimation risk means the form is right but the parameters are imprecise because they came from a finite sample; this error is a variance that shrinks as the sample size T grows, so more data does help. A useful exam line is that a simpler model carries less estimation risk but more misspecification risk, and a richer model the reverse, which is exactly the bias-variance trade-off that AIC and BIC try to balance.
If a VaR model produces more violations than expected, does that mean it overestimates or underestimates risk?
It underestimates risk. The expected number of violations over T₀ days is T₀ times p, so more violations than that means the observed violation rate π̂ is greater than p. That happens when the VaR threshold is too small, letting losses breach it too often - so the model is understating how bad losses can be, and capital set from it would be too low. The opposite case, too few violations (π̂ below p), means the VaR is too large and risk is overstated: safe, but capital-inefficient. Getting this direction the wrong way round is one of the most common mistakes in backtesting questions.
Can AI help me with model risk and VaR backtesting in ETF5952?
Yes, as a study aid for the method. You can ask Sia to explain the ideas step by step - the difference between misspecification and estimation risk, how AIC and BIC penalise complexity, how to build a hit sequence, or how the Kupiec statistic is formed and compared with a chi-square critical value - and to walk through a practice backtest so you can reproduce the working yourself. Sia explains and coaches the technique, including the all-important direction of a rejection; it does not sit your quizzes, assignments or the closed-book final exam for you, and you should always check formulas and the Basel figures against your own Moodle materials.
Exam move
Split your revision into the two halves this topic tests. First, be able to state the two sources of model risk in a sentence each - misspecification is the wrong form (a bias more data cannot cure), estimation is imprecise parameters (a variance that shrinks with T) - and to write the AIC and BIC formulas, remembering that you minimise them and that BIC is the stricter penalty so it favours simpler models. Second, drill the backtesting workflow end to end on a short numerical example: count the violations, form the expected count T₀ times p and the observed rate, state the hypotheses, compute the Kupiec likelihood ratio, compare it with the chi-square (1 df) critical value 3.841, and then interpret the direction - too many violations means the model understates risk. Practise the direction words out loud, because a correct statistic with the wrong 'over/under-states risk' reading earns nothing. Keep the coverage tests straight: unconditional checks the rate, independence checks the timing (clustering), and conditional checks both with 2 degrees of freedom. Finally, memorise the Basel facts exactly - 2.5% ES sets capital while 1% and 2.5% VaR are backtested against the prior 12 months - and remember ES is always at least VaR. The final exam is closed book and worth 40% of the unit; the exam duration is not published in the unit materials, so plan your time in proportion to the marks on each question and confirm the exact date and length on Moodle or the Monash exam timetable.
Working through Model Risk and VaR Backtesting in ETF5952? Sia is AskSia’s AI Finance tutor — ask any ETF5952 Model Risk and VaR Backtesting question and get a clear, step-by-step explanation grounded in how ETF5952 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
