DATA4207 · Data Analysis in the Social Sciences
Model Selection, Assumptions and Diagnostics
Week 3 works through choosing and specifying regression models — how the model's goal and the way data was collected shape which variables and confounders to include — and how to check the assumptions OLS rests on using diagnostic plots. It covers uncertainty, sample size and power analysis, and knitting reproducible R Markdown for group tasks. Diagnosing a model honestly is exactly what markers look for in the methods and results sections of every project.
What this chapter covers
- 01Model selection and specification: the model's goal and the data-collection design constrain which variables to include
- 02How data is collected: observational vs experimental (randomised) studies; RCTs eliminate confounding
- 03Total survey error = random error + systematic error (bias); measurement, non-response, selection and frame error
- 04Ordinary Least Squares: lm() minimises Σ(yᵢ − ŷᵢ)²; residuals = observed − predicted
- 05Checking the five OLS assumptions with diagnostic plots: residuals-vs-fitted, normal Q-Q, residual-vs-leverage
- 06Multicollinearity via VIF and correlation matrices; outliers via Cook's distance
- 07Uncertainty, sample size and running a power analysis
- 08Theory-driven vs utility/machine-assisted (LASSO) variable selection — 'no right answer, but wrong ones'
Diagnosing three regression problems from the output
- +2(a) A widening funnel in residuals-vs-fitted means the residual variance grows with the fitted value — non-constant variance, so the homoscedasticity assumption is violated. Fix: transform the outcome (log or square-root) to stabilise variance, or use heteroscedasticity-robust standard errors.
- +2(b) A Q-Q plot that bends at both tails means the residuals have heavier tails than a normal distribution — the normality-of-residuals assumption is violated. Fix: transform the outcome or an offending predictor, remove/justify outliers, or rely on a large sample where OLS inference is more robust.
- +2(c) A VIF well above the common 5-10 rule of thumb (here ≈ 8.5) signals multicollinearity — the two predictors are highly correlated, inflating their standard errors. Fix: drop one, combine them into an index, or collect data that separates them. (Confirm the threshold your unit uses on Canvas.)
Key terms
- Ordinary Least Squares (OLS)
- The estimation method lm() uses: it finds the line or hyperplane that minimises the sum of squared residuals, Σ(yᵢ − ŷᵢ)². Squaring stops positive and negative errors cancelling and penalises large errors more heavily.
- Residual
- For each observation, observed minus predicted value (yᵢ − ŷᵢ) — how far off the model's prediction was. Diagnostic plots of residuals are the main tool for checking OLS assumptions.
- Variance inflation factor (VIF)
- A measure of how much a predictor's variance is inflated by correlation with the other predictors. High values flag multicollinearity; a common rule of thumb treats values above 5-10 as a concern (confirm the exact cut-off on Canvas).
- Cook's distance
- A leverage-based influence measure that flags observations whose removal would substantially change the fitted model. Used alongside standardised residuals to identify outliers.
- Total survey error
- The difference between observed and true values, decomposed into random error (reduced by larger samples) and systematic error / bias (from design or data-collection flaws, such as non-response, selection or measurement error).
- Power analysis
- A calculation of the sample size needed to detect an effect of a given size with acceptable probability, linking uncertainty to how much data you should collect.
Model Selection, Assumptions and Diagnostics FAQ
How do I check the OLS assumptions?
Fit the model, then read the standard diagnostic plots: residuals-vs-fitted for linearity and homoscedasticity (you want a flat, random cloud), a normal Q-Q plot for normal residuals (points near the line), and a residual-vs-leverage plot with Cook's distance for influential outliers. Check multicollinearity separately with VIFs and a correlation matrix. Serious violations mean biased estimates or unreliable predictions, so you either fix them or discuss them honestly.
Why is variable selection so hard — is there a right answer?
There is no single correct model, but there are wrong ones. The unit favours theory-driven selection — choose variables because a mechanism justifies them — sometimes mixed with utility or machine-assisted approaches like LASSO, depending on the model's purpose. The key is to justify your choices; a report that maximises R² by throwing in everything usually scores worse than one that argues for a smaller, defensible set.
What is the difference between random and systematic error?
Random error is naturally occurring noise driven by precision limits, and you reduce it by collecting more data. Systematic error is bias from design or collection flaws — non-response, selection, measurement or frame error — and more data does not fix it; you have to understand and mitigate the source. Standard errors quantify only the random component, not the systematic one.
Can AI help me with regression diagnostics in DATA4207?
Yes, as a study aid. Sia can explain what each diagnostic plot shows, help you name the violated assumption, and suggest defensible fixes and variable-selection reasoning, step by step. It teaches the method and checks your thinking; it does not do graded work, and University of Sydney academic-integrity rules apply — confirm on Canvas whether AI is permitted for a task.
Assessment move
Make diagnostics a habit, not an afterthought. For any model you fit this week, immediately produce the residuals-vs-fitted, Q-Q and leverage plots and practise saying in one sentence which assumption each supports or challenges and what you would do about it — that narration is exactly what the methods section of a group project or the individual report rewards. Learn to compute and read VIFs and to spot influential points with Cook's distance. Rehearse justifying a small, theory-driven set of predictors over a kitchen-sink model. Since group work is submitted as R Markdown that must knit unchanged, get the knit-to-PDF workflow solid now. Confirm any unit-specific thresholds (such as the VIF cut-off) and lab requirements on Canvas.
Working through Model Selection, Assumptions and Diagnostics in DATA4207? Sia is AskSia’s AI Statistics tutor — ask any DATA4207 Model Selection, Assumptions and Diagnostics question and get a clear, step-by-step explanation grounded in how DATA4207 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
