QBUS2820 Chap.6 Model and Variable Selection
Model and Variable Selection
Model selection chooses among fitted structures, while variable selection determines which predictors enter those structures. This chapter compares AIC, BIC and K-fold cross-validation, then examines best-subset, forward-stepwise and backward-stepwise searches. Each tool separates candidate generation from the selection criterion.
Lower information-criterion values or lower held-out loss identify the preferred candidate only when models use a comparable response, dataset and likelihood basis, and when preprocessing and search are repeated inside the validation boundary. Information criteria and cross-validation should not be treated as interchangeable labels.
AIC and BIC derive from likelihood and impose explicit complexity penalties, while cross-validation estimates the selected loss through held-out predictions. Their rankings can differ because they optimise related but distinct aims. Candidate models must be fitted to the same response cases for criterion values to be compared directly.
Search algorithms also constrain what can win: best-subset explores broadly, while forward and backward procedures follow paths that can miss useful combinations. Transformations and interactions enlarge the candidate universe and should be recorded before selection. Repeating a stepwise search inside each fold is computationally heavier but necessary for an honest predictive estimate.
If inference follows a search, ordinary standard errors and p-values omit selection uncertainty. Stability tables showing how often variables appear can be more informative than presenting one chosen subset as inevitable. Domain requirements can keep variables in a model even when a purely numerical search would remove them.
What this chapter covers
- 01
Candidate model definition
- 02
AIC penalties
- 03
BIC penalties
- 04
K-fold cross-validation
- 05
Best-subset selection
- 06
Forward stepwise search
- 07
Backward stepwise search
- 08
Post-selection uncertainty
Worked example · free
Compare AIC and BIC choices
- +1The marks beside these steps are a revision structure, not a published University marking scheme. Model 1 AIC is 240 + 2 × 3 = 246.
- +1Model 2 AIC is 232 + 2 × 6 = 244.
- +1Model 3 AIC is 228 + 2 × 10 = 248.
- +1AIC selects Model 2 because 244 is smallest.
- +1BIC uses log(100) per parameter, a stronger penalty than two, so it can select the smaller Model 1.
Key terms
- AIC
- An information criterion combining likelihood fit with a penalty of two per parameter.
- BIC
- An information criterion using a log-sample-size penalty that grows stronger with n.
- Cross-Validation
- Rotating held-out folds to estimate predictive loss for model choice.
- Best Subset
- A search that evaluates every candidate predictor subset when computationally feasible.
- Forward Selection
- A greedy search that starts small and adds one variable at a time.
- Backward Selection
- A greedy search that starts large and removes one variable at a time.
- Candidate Universe
- The full set of predictor combinations, transformations and interactions that a selection procedure is permitted to consider.
Model and Variable Selection FAQ
Should I always choose the smallest AIC?
Choose the smallest AIC only among genuinely comparable candidate fits. If prediction is the operational target, confirm the choice with an honest held-out design and check whether the gain is stable.
Why can AIC and BIC disagree?
They apply different complexity penalties. BIC's per-parameter penalty grows with sample size, so it often prefers smaller models than AIC when added terms improve likelihood only modestly.
What is the limitation of stepwise search?
Stepwise procedures follow greedy paths and can miss useful combinations whose variables do not look strong one at a time. Their selected model also inherits uncertainty from the search process.
Can required business variables stay in every candidate?
Yes. A modelling plan can force essential controls or operational fields into all candidates while selection acts on the remaining features. Record that constraint so the search and interpretation are reproducible.
How should near-tied models be handled?
Inspect fold variation, stability and deployment cost rather than declaring a precise winner from a tiny mean difference. A simpler candidate can be preferable when predictive evidence is effectively tied.
Exam move
Create a selection worksheet with one row per candidate and columns for fit, parameter count, AIC, BIC and cross-validated loss. Recompute simple examples so that signs and penalties become automatic. Draw five cross-validation rows and shade a different held-out fold in each row, then explain where scaling and feature selection occur.
Compare best-subset, forward and backward searches on coverage, computation and path dependence. Use an original example where two variables matter only together to show why a greedy path can fail. When interpreting a selected model, remember that ordinary post-selection p-values ignore the search. Write a final recommendation that names the candidate set, search procedure, criterion, resampling unit and stability evidence.
The aim is a reproducible decision rule, not a collection of minima detached from how the candidates were produced.
Working through Model and Variable Selection in QBUS2820? Sia is AskSia’s AI Statistics tutor — ask any QBUS2820 Model and Variable Selection question and get a clear, step-by-step explanation grounded in how QBUS2820 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
