MTH2021 · Linear Algebra with Applications
Singular Value Decomposition
Week 12 extends diagonalisation to every matrix — rectangular or singular — via the singular value decomposition A = UΣVᵀ. The singular values σᵢ = √(eigenvalues of AᵀA) sit on Σ; the columns of V are eigenvectors of AᵀA and uᵢ = Avᵢ/σᵢ build U. The SVD yields the Moore–Penrose pseudoinverse A⁺ and the unique minimal-length least-squares solution. As the unit's capstone, it is a headline topic of the comprehensive 50% final.
What this chapter covers
- 01AᵀA is symmetric with nonnegative eigenvalues; singular values σᵢ = √λᵢ of AᵀA
- 02A matrix of rank r has exactly r positive singular values
- 03SVD A = UΣVᵀ: U (m×m orthogonal), Σ (m×n nonnegative diagonal), V (n×n orthogonal)
- 04Columns of V = orthonormal eigenvectors of AᵀA (ordered by decreasing σ); uᵢ = Avᵢ/σᵢ for i ≤ r; extend U with a coker(A) basis
- 05The singular vectors give orthonormal bases of the four fundamental subspaces
- 06Singular value expansion A = σ₁u₁v₁ᵀ + … + σᵣuᵣvᵣᵀ (reduced SVD)
- 07Pseudoinverse of Σ: Σ⁺ with 1/σᵢ on the diagonal; A⁺ = VΣ⁺Uᵀ; A⁺ = A⁻¹ when A is invertible
- 08Ax = b has a unique minimal-length least-squares solution x = A⁺b (lying in row(A))
Building the SVD of a 3×2 matrix
- +1Form AᵀA. AᵀA = [[1·1 + 0 + 1·1, 1·0 + 0 + 1·1], [0·1 + 1·0 + 1·1, 0 + 1 + 1]] = [[2, 1], [1, 2]].
- +1Eigenvalues/eigenvectors of AᵀA = [[2,1],[1,2]]: characteristic polynomial (2−λ)² − 1 = (λ − 3)(λ − 1), so λ = 3, 1 with eigenvectors (1, 1) and (1, −1). Order by decreasing eigenvalue.
- +1Singular values and V: σ₁ = √3, σ₂ = √1 = 1. Normalise the eigenvectors into V's columns: v₁ = (1/√2)(1, 1), v₂ = (1/√2)(1, −1). So V = (1/√2)[[1, 1], [1, −1]] and Σ = [[√3, 0], [0, 1], [0, 0]] (3×2).
- +1Left singular vectors uᵢ = Avᵢ/σᵢ. Av₁ = A(1,1)ᵀ/√2 = (1, 1, 2)ᵀ/√2, so u₁ = (1,1,2)ᵀ/(√2·√3) = (1, 1, 2)ᵀ/√6. Av₂ = A(1,−1)ᵀ/√2 = (1, −1, 0)ᵀ/√2, so u₂ = (1, −1, 0)ᵀ/√2. Both are unit vectors (1+1+4 = 6; 1+1 = 2).
- +1Extend U with a basis of coker(A) = ker(Aᵀ). Solving Aᵀw = 0 gives w = (1, 1, −1)ᵀ, normalised u₃ = (1, 1, −1)ᵀ/√3. Then A = UΣVᵀ with U = [u₁ u₂ u₃], Σ = diag-echelon(√3, 1), V as above.
Key terms
- Singular value
- σᵢ = √λᵢ, the square root of an eigenvalue of the symmetric matrix AᵀA; there are exactly rank(A) positive singular values.
- Singular value decomposition
- A = UΣVᵀ with U (m×m) and V (n×n) orthogonal and Σ (m×n) diagonal of nonnegative singular values; every matrix has one.
- Right / left singular vectors
- The columns of V (eigenvectors of AᵀA) and of U (eigenvectors of AAᵀ, or uᵢ = Avᵢ/σᵢ); they give orthonormal bases of the four fundamental subspaces.
- Singular value expansion
- A = σ₁u₁v₁ᵀ + … + σᵣuᵣvᵣᵀ, the reduced SVD as a sum of rank-one terms weighted by the singular values.
- Moore–Penrose pseudoinverse
- A⁺ = VΣ⁺Uᵀ, where Σ⁺ has 1/σᵢ on the diagonal; it equals A⁻¹ when A is invertible and generalises inversion otherwise.
- Minimal-length least-squares solution
- x = A⁺b, the unique least-squares solution of Ax = b of smallest norm; it lies in the row space of A.
Singular Value Decomposition FAQ
How do I actually compute an SVD?
Start from AᵀA: find its orthonormal eigenvectors and order them by decreasing eigenvalue to form V, and take σᵢ = √λᵢ for Σ. Then set uᵢ = Avᵢ/σᵢ for each positive singular value, and if A is not square (or is rank-deficient) complete U with an orthonormal basis of ker(Aᵀ). Assemble A = UΣVᵀ.
Why does the SVD exist for every matrix when diagonalisation does not?
Because it is built from AᵀA, which is symmetric with nonnegative eigenvalues, so it always has an orthonormal eigenbasis by the Spectral Theorem — no matter what shape or rank A has. Diagonalisation needs A itself to have enough eigenvectors, which can fail; the SVD sidesteps that by working with AᵀA and AAᵀ instead.
What is the pseudoinverse and when is it just the inverse?
The Moore–Penrose pseudoinverse A⁺ = VΣ⁺Uᵀ inverts the nonzero singular values (Σ⁺ carries 1/σᵢ) and zeroes the rest. When A is square and invertible, A⁺ coincides exactly with A⁻¹; otherwise it is the natural generalisation used to solve least-squares problems.
How does the SVD solve least squares?
For any b, x = A⁺b is the least-squares solution of Ax = b of minimal length, and it lies in the row space of A. So the SVD not only gives the best fit (as the normal equations do) but singles out the smallest such solution when there are infinitely many — a sharper result than (AᵀA)⁻¹Aᵀb, which needs independent columns.
Where is the SVD examined?
As the Week-12 capstone it is assessed in its weekly quiz and, above all, in the comprehensive 50% final, where a full A = UΣVᵀ construction (often with the pseudoinverse) is a headline computation. Because it ties together AᵀA, orthogonal diagonalisation and the four subspaces, it is a favourite synthesis question.
Exam move
Fix the pipeline: AᵀA → orthonormal eigenvectors (ordered by decreasing eigenvalue) = V, σᵢ = √λᵢ → Σ, uᵢ = Avᵢ/σᵢ, then complete U with a ker(Aᵀ) basis. Check every uᵢ is a unit vector and, if asked, form A⁺ = VΣ⁺Uᵀ for the minimal-length least-squares solution. Because the SVD synthesises symmetric diagonalisation and the four subspaces, rehearse it as a full construction. It is a headline topic of the 50% final. When your U columns aren't orthonormal, ask Sia to recheck uᵢ = Avᵢ/σᵢ and the coker extension.
Working through Singular Value Decomposition in MTH2021? Sia is AskSia’s AI Mathematics tutor — ask any MTH2021 Singular Value Decomposition question and get a clear, step-by-step explanation grounded in how MTH2021 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
