MTH2021 · Linear Algebra with Applications
Gram-Schmidt, Projection & Least Squares
Weeks 9–10 turn any basis into an orthonormal one via the Gram–Schmidt process, project vectors onto subspaces with the projection theorem, and solve overdetermined systems by least squares. The normal equations AᵀAx = Aᵀb (always consistent) give the best fit, and the same idea fits lines and polynomials to data. The Fundamental Theorem of Linear Algebra (row(A)^⊥ = ker(A)) ties it together. This is heavily examined in the comprehensive 50% final.
What this chapter covers
- 01Orthogonal and orthonormal sets; an orthogonal set of nonzero vectors is automatically linearly independent
- 02Gram–Schmidt process: v₁ = u₁, then vₖ = uₖ − Σ_{i
- 03Projection Theorem: every u = w + w^⊥ with projᵥᵥ u = Σⱼ ⟨u, vⱼ⟩ vⱼ for an orthonormal basis of W
- 04V = W ⊕ W^⊥; every finite-dimensional inner product space has an orthonormal basis
- 05Fundamental Theorem of Linear Algebra: row(A)^⊥ = ker(A); ℝⁿ = row(A) ⊕ ker(A), ℝᵐ = col(A) ⊕ coker(A)
- 06Closest-point / least-squares problem: minimise ‖b − Ax‖; the error vector is b − Ax
- 07Normal equations AᵀAx = Aᵀb are always consistent; AᵀA invertible ⇔ A has independent columns
- 08Data fitting: line y = a + bx and polynomial fits via MᵀMv = Mᵀy
Gram–Schmidt in ℝ² producing an orthonormal basis
- +1Take v₁ = u₁ = (1, −3). Its squared norm is ‖v₁‖² = 1² + (−3)² = 1 + 9 = 10.
- +1Compute the projection coefficient: ⟨u₂, v₁⟩ = (2)(1) + (2)(−3) = 2 − 6 = −4, so ⟨u₂, v₁⟩/‖v₁‖² = −4/10 = −0.4.
- +1Subtract the projection: v₂ = u₂ − (−0.4)v₁ = (2, 2) + 0.4·(1, −3) = (2.4, 0.8). Check orthogonality: ⟨v₂, v₁⟩ = 2.4·1 + 0.8·(−3) = 2.4 − 2.4 = 0 ✓. Note (2.4, 0.8) = 0.8·(3, 1), so v₂ ∝ (3, 1).
- +1Normalise. ‖v₁‖ = √10 and ‖(3,1)‖ = √10, so e₁ = (1/√10)(1, −3) and e₂ = (1/√10)(3, 1). These are orthonormal: each has length 1 and ⟨e₁, e₂⟩ = (1·3 + (−3)·1)/10 = 0 ✓.
Key terms
- Orthonormal set
- A set of mutually orthogonal unit vectors; any orthogonal set of nonzero vectors is linearly independent, and normalising gives an orthonormal one.
- Gram–Schmidt process
- An algorithm that converts a basis {u₁, …, uₙ} into an orthogonal (then orthonormal) basis by vₖ = uₖ − Σ_{i
- Orthogonal projection
- projᵥᵥ u = Σⱼ ⟨u, vⱼ⟩vⱼ for an orthonormal basis of W; it is the closest point of W to u, and u − projᵥᵥ u lies in W^⊥.
- Projection Theorem
- Every vector u splits uniquely as u = w + w^⊥ with w ∈ W and w^⊥ ∈ W^⊥; equivalently V = W ⊕ W^⊥.
- Normal equations
- AᵀAx = Aᵀb, the always-consistent system whose solutions are the least-squares solutions of Ax = b; AᵀA is invertible iff A has linearly independent columns.
- Fundamental Theorem of Linear Algebra
- The orthogonality relations row(A)^⊥ = ker(A) and col(A)^⊥ = coker(A), giving ℝⁿ = row(A) ⊕ ker(A) and ℝᵐ = col(A) ⊕ coker(A).
Gram-Schmidt, Projection & Least Squares FAQ
Do I normalise during Gram–Schmidt or at the end?
At the end. Run the subtraction step vₖ = uₖ − Σ (⟨uₖ, vᵢ⟩/‖vᵢ‖²)vᵢ with the un-normalised orthogonal vectors, checking orthogonality after each one, and only divide by the norms once all the vₖ are found. Normalising midway is allowed but just multiplies the arithmetic.
Why are the normal equations always solvable?
Because AᵀAx = Aᵀb is exactly the condition that Ax equals the orthogonal projection of b onto col(A), and that projection always exists. So even when Ax = b has no solution, the least-squares problem does. If A has independent columns, AᵀA is invertible and the solution x = (AᵀA)⁻¹Aᵀb is unique.
What is least squares actually minimising?
The length of the error vector ‖b − Ax‖. Geometrically the best x makes Ax the point of col(A) closest to b, so the error b − Ax is orthogonal to col(A) — which is precisely Aᵀ(b − Ax) = 0, i.e. the normal equations. That is why projection and least squares are the same idea.
How does line/polynomial fitting use this?
You write the model as Mv = y, where each row of M holds the basis functions evaluated at a data point (for a line, rows [1, xᵢ]; for a polynomial, a Vandermonde row). Then you solve the normal equations MᵀMv = Mᵀy for the coefficients v. The result is the best-fit line or curve in the least-squares sense.
How central is this to the exam?
Very. Weeks 9–10 land after Test 2's window, so Gram–Schmidt, projection and least squares are assessed mainly in the comprehensive 50% final, and they are among its most reliable computation-and-proof topics. Expect both a numerical fit and a projection-theorem or Fundamental-Theorem argument.
Exam move
Drill Gram–Schmidt with the un-normalised orthogonal vectors, checking ⟨vₖ, vᵢ⟩ = 0 after every step and normalising only at the end. For least squares, always build AᵀA and Aᵀb first and solve the small square system rather than the tall one, and practise the line-of-best-fit setup Mv = y → MᵀMv = Mᵀy. Rehearse the projection-theorem and Fundamental-Theorem-of-Linear-Algebra proofs, since these are exam-favoured. This material is examined mainly in the 50% final. When a Gram–Schmidt vector fails the orthogonality check, ask Sia to trace the projection subtraction.
Working through Gram-Schmidt, Projection & Least Squares in MTH2021? Sia is AskSia’s AI Mathematics tutor — ask any MTH2021 Gram-Schmidt, Projection & Least Squares question and get a clear, step-by-step explanation grounded in how MTH2021 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
