QBUS2820 Chap.4 K-Nearest Neighbours Regression
K-Nearest Neighbours Regression
K-nearest neighbours regression predicts locally by averaging the responses attached to the closest training inputs. The method avoids imposing one global linear equation, but its success depends on how distance is defined, whether predictors are scaled, how K is tuned and how densely the predictor space is populated.
This chapter develops the flexibility trade-off, Mahalanobis distance, the curse of dimensionality, computational considerations and the contrast between local nonparametric prediction and global parametric regression. The neighbour search is an entire fitted pipeline rather than a single averaging step.
Missing-value treatment, transformations, scaling and feature selection all change the geometry and must be learned from training cases. A validation split then compares K values using the same prepared predictor space. Ties need a declared rule, particularly for classification, and repeated identical inputs can deserve special attention because their outcomes reveal local noise.
Weighted KNN gives closer neighbours more influence, but the weighting rule adds another tuning choice rather than removing the need for evaluation. Computational cost is deferred to prediction time because distances to training observations must be calculated or approximated for each target.
Data structures can accelerate search in modest dimensions, while high-dimensional spaces weaken both speed and meaningful distance contrast. KNN predictions stay within local response averages and therefore avoid extreme extrapolation, but that same property prevents them from extending a genuine trend beyond the observed cloud. Always inspect the target's distance to its neighbours.
What this chapter covers
- 01
Neighbour averaging at a target
- 02
Euclidean distance and scaling
- 03
Mahalanobis distance
- 04
Small and large K
- 05
Cross-validated K selection
- 06
Curse of dimensionality
- 07
Computational prediction cost
- 08
KNN compared with linear regression
Worked example · free
Average three delivery neighbours
- +1The marks in this worked drill are not a University assessment scheme; they only divide the practice method. Confirm the neighbour search used predictors on comparable standardised scales.
- +1Add the responses: 31 + 35 + 40 = 106 minutes.
- +1Divide by K = 3 to obtain 35.33 minutes.
- +1A smaller K would make the prediction more local and variable.
- +1A larger K would smooth more strongly and can raise bias; select K with held-out error.
Key terms
- Nearest Neighbour
- A training case whose predictors are closest to the target under the chosen distance.
- Euclidean Distance
- Straight-line distance in predictor space, sensitive to measurement scale.
- Mahalanobis Distance
- Distance adjusted for predictor covariance and correlation structure.
- Neighbourhood Size
- The tuning value K controlling how many responses enter a local prediction.
- Dimensionality Curse
- The sparsity and weakened distance contrast that arise as predictor count grows.
- Nonparametric Method
- A method that does not impose one fixed finite-dimensional response equation.
- Distance Weighting
- A KNN variant that gives closer neighbours greater influence through a declared distance-based weighting rule.
K-Nearest Neighbours Regression FAQ
Why must predictors be scaled?
Distance depends on numerical units. Without scaling, a large-unit predictor can dominate the neighbour search even when it is not more informative. Fit scaling values on training cases only.
What happens when K is small?
The prediction follows a very local group and can capture fine structure, but it becomes sensitive to sample noise. This usually lowers bias while raising variance.
When can linear regression beat KNN?
A stable linear relationship, many predictors or sparse data can favour linear regression. KNN needs enough nearby observations and its flexibility becomes expensive in high-dimensional spaces.
Can closer neighbours receive more weight?
Yes. Distance-weighted KNN can give nearby cases greater influence, but the weighting function and K become tuning choices that require the same honest validation as an unweighted average.
How can I recognise unsafe extrapolation?
Compare the target with the training cloud and record neighbour distances. If every selected neighbour is far away or lies on one side, the local average has weak support for that target.
Exam move
Draw a target point and expand a circle until it contains K training cases. Use that picture to narrate the algorithm: scale predictors, compute distances, sort cases, select the nearest responses and average them. Work several tiny examples by hand so the membership and mean are both visible. Then vary K without changing the data and explain how the neighbourhood, prediction and boundary change.
Compare Euclidean and covariance-adjusted distance conceptually rather than treating Mahalanobis distance as a magic fix. Create a table contrasting KNN and linear regression on locality, interpretability, computational cost, dimensionality and extrapolation. Finish with a resampling plan that chooses K using held-out error while fitting every scaling transformation inside each training fold.
The objective is not merely to calculate an average; it is to justify why the selected neighbours represent the target decision.
Working through K-Nearest Neighbours Regression in QBUS2820? Sia is AskSia’s AI Statistics tutor — ask any QBUS2820 K-Nearest Neighbours Regression question and get a clear, step-by-step explanation grounded in how QBUS2820 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
