CVEN9405 Chap.7 Logit Estimation, Validation and Forecasting
Logit Estimation, Validation and Forecasting
Estimation, validation and forecasting are different claims. Maximum likelihood chooses parameters that make observed choices relatively likely under a specified model. Model comparison tests restrictions and weighs better fit against parsimony and behavioural plausibility.
Validation challenges predictions with evidence not used to estimate the parameters, while forecasting transfers the relationship to a changed network, population or policy. This chapter uses log-likelihood differences, likelihood-ratio tests, coefficient signs, implied behavioural values and holdout performance as complementary checks.
It also shows why aggregate accuracy can conceal offsetting spatial or demographic errors and why every published scenario needs a reproducible audit trail.
What this chapter covers
- 01
Likelihood and log-likelihood for observed choices
- 02
Restricted and unrestricted model comparison
- 03
Likelihood-ratio statistics and degrees of freedom
- 04
Coefficient signs, magnitude and implied value of time
- 05
Independent validation across people, places or periods
- 06
Forecast transfer, behavioural unit tests and audit trails
Worked example · free
Compare two nested mode-choice specifications
- +1Log-likelihood improvement = -505.9 - (-512.8) = 6.9.
- +1LR = 2 × 6.9 = 13.8.
- +1Use three degrees of freedom because the unrestricted model adds three independent parameters.
- +1Since 13.8 exceeds 7.815, reject the restrictions at the supplied threshold, then inspect signs and forecast consequences.
Key terms
- Log-Likelihood
- The sum of log predicted probabilities assigned to observed choices under an estimated model.
- Restricted Model
- A model that imposes specified parameter equalities or exclusions within a more general nested specification.
- Likelihood-Ratio Test
- A test comparing nested models through twice their log-likelihood difference and the number of restrictions.
- Holdout Validation
- Evaluation on observations deliberately excluded from model fitting or specification tuning.
- Behavioural Unit Test
- A controlled input change used to verify that a model output moves in the expected direction and remains coherent.
Logit Estimation, Validation and Forecasting FAQ
Why is better in-sample fit not enough for forecasting?
Adding parameters can improve fit to estimation records while capturing noise or unstable relationships. Forecast use also requires sensible signs and magnitudes, predictor availability, performance on new observations and credible response under policy changes. Fit is evidence, not a transfer guarantee.
What does a likelihood-ratio test compare?
It compares nested specifications estimated on the same observations and choice context. The statistic measures twice the log-likelihood improvement from relaxing restrictions, with degrees of freedom equal to the number of independent restrictions. It does not compare arbitrary non-nested models directly.
How can aggregate validation hide a serious error?
Over-prediction in one zone or group can cancel under-prediction elsewhere, preserving a regional total. Planning decisions often depend on location, time or segment, so validation should examine those scales alongside aggregate shares and totals.
Why should coefficient signs be checked before celebrating model fit?
A coefficient with an implausible direction may indicate miscoding, confounding, poor variation or a misspecified choice set. Such a model can match observed shares yet respond perversely to a policy forecast. Compare every sign and scale with behavioural expectations, data units and related specifications before relying on likelihood or pseudo-fit measures.
What is the practical value of a holdout validation sample?
A holdout sample tests predictions on observations that did not determine the estimated coefficients. It offers a more honest indication of generalisation than rechecking the estimation data, although it cannot reproduce every future policy change. Preserve the choice-set construction and availability rules, then compare shares, log loss and segment-level errors with useful benchmarks.
How should a mode-share forecast be reported when inputs are uncertain?
Present the central forecast together with scenarios for the inputs that matter most, such as travel time, fare, parking cost or service availability. Keep coefficients fixed unless the scenario explicitly changes preferences. Explain which shares move, which alternatives gain those travellers and whether the resulting elasticities remain behaviourally credible.
Exam move
Keep three headings on every model review page: estimation, validation and forecast. Under estimation, record the sample, choice set, specification, coefficient signs and log-likelihood. Under validation, record what evidence was held out and examine totals, spatial patterns, segments and directional response. Under forecast, list changed inputs, variables outside the observed range and assumptions about parameter stability.
Practise likelihood-ratio arithmetic with negative log-likelihood values until the subtraction is automatic, and always state the number of restrictions. Pair formal tests with behavioural checks such as implied value of time and plausible substitution. Run simple unit tests by increasing cost or time for one mode and confirming its probability falls while all shares still sum to one.
For trip-generation models, test minimum and maximum predictor values and inspect residual structure. Finish by writing an audit trail containing input version, coefficient set, availability rules, scenario assumptions, validation metrics and post-processing. A reproducible limitation is more useful than an unsupported claim of accuracy.
