MAST10007 Chap.13 Least Squares, Orthogonal Matrices, Symmetry and SVD
Least Squares, Orthogonal Matrices, Symmetry and SVD
Chapter 13 develops least squares, orthogonal matrices, symmetry and svd for University of Melbourne MAST10007. Least squares chooses Ax closest to b, so the residual b-Ax is orthogonal to the column space. That condition yields the normal equations. Orthogonal matrices preserve dot products and have inverse equal to transpose. Real symmetric matrices admit orthonormal eigenbases.
The SVD extends this geometry to every real matrix: V chooses input directions, Σ scales them, and U places the outputs. It pairs exact definitions with a new completed example, two concept diagrams, diagnostic contrasts and answered retrieval prompts.
These topics share an orthogonality viewpoint.
Least squares replaces an inconsistent target with its projection onto the column space and checks the residual against every column. Orthogonal matrices preserve the geometry used by that check, while symmetric matrices permit diagonalisation with an orthonormal eigenbasis.
The singular value decomposition separates input directions, nonnegative scaling, and output directions even when the original matrix is not square or diagonalizable in the eigenvalue sense. A complete solution labels those roles and verifies the defining relation or orthogonality condition.
What this chapter covers
- 01
Inconsistent systems and best fit
- 02
Orthogonal residuals
- 03
Normal equations
- 04
Projection viewpoint
- 05
Orthogonal matrices
- 06
Length and angle preservation
- 07
Symmetric matrices
- 08
Orthogonal diagonalisation
- 09
Singular value decomposition
- 10
Unit circle to ellipse
- 11
Residual checks against the column space
- 12
Orthonormal eigenbases for symmetric matrices
- 13
Input, scale and output roles in the SVD
Least Squares, Orthogonal Matrices, Symmetry and SVD worked example
- stepUse A with rows (1,0),(1,1),(1,2) and b=(1,2,2). Then A^TA=[[3,3],[3,5]] and A^Tb=(5,6).
- stepSolve 3a+3b=5 and 3a+5b=6. Subtraction gives 2b=1, so b=1/2.
- stepThen 3a+3/2=5, giving a=7/6. The fitted line is y=7/6+x/2.
- stepThe residual is orthogonal to both columns of A, which checks the intercept and slope equations without pretending the inconsistent data are fitted exactly.
Key terms
- Inconsistent systems and best fit
- When Ax=b is inconsistent, least squares chooses x̂ minimising ||b−Ax||. It does not make the equations true; it makes the residual as short as possible. The fitted vector A x̂ lies in Col(A).
- Orthogonal residuals
- At the optimum the residual r=b−A x̂ is orthogonal to every column of A, so A^Tr=0. This geometric condition is the heart of least squares and provides a direct verification after solving.
- Normal equations
- Expanding A^T(b−A x̂)=0 gives A^TA x̂=A^Tb. These normal equations are square even when A is rectangular. Uniqueness requires independent columns, not merely that the equations can be written.
- Projection viewpoint
- The fitted vector is the orthogonal projection of b onto Col(A). If the columns are orthonormal, coefficients are immediate inner products; otherwise the normal equations compute the projection coordinates.
- Orthogonal matrices
- A real square matrix Q is orthogonal when Q^TQ=I. Its inverse is Q^T, and its columns form an orthonormal basis. Orthogonal matrices preserve lengths, dot products, angles, and absolute volume.
- Length and angle preservation
- Because ||Qx||²=x^TQ^TQx=||x||², an orthogonal transformation cannot stretch. Its determinant is ±1. In two dimensions the sign distinguishes rotation-type orientation from reflection-type orientation.
- Symmetric orthogonal diagonalisation
- A real symmetric matrix has real eigenvalues and an orthonormal basis of eigenvectors. Its diagonalisation can therefore use an orthogonal change of basis, so the inverse of the eigenvector matrix is its transpose.
- Singular value decomposition
- The SVD factors a real matrix into orthogonal input directions, nonnegative singular-value scaling, and orthogonal output directions. It applies to rectangular and rank-deficient matrices and makes stretching and collapsed directions geometrically visible.
Least Squares, Orthogonal Matrices, Symmetry and SVD FAQ
How do I check inconsistent systems and best fit?
When Ax=b is inconsistent, least squares chooses x̂ minimising ||b−Ax||. It does not make the equations true; it makes the residual as short as possible. The fitted vector A x̂ lies in Col(A). Q^T.
What is the main trap in least squares, orthogonal matrices, symmetry and svd?
Do not write A^{-1}b for a rectangular least-squares problem. Normal equations express orthogonality, not exact solvability. Singular values are nonnegative and are not generally eigenvalues of A.
Can AI help with this MAST10007 topic?
Yes. Ask Sia to explain one step, create a fresh practice problem, or check your reasoning. Use it to learn rather than complete graded assessment, and follow University of Melbourne academic-integrity rules.
Why do the normal equations express a closest-point condition?
At the least-squares solution, the fitted vector lies in the column space and the remaining residual is perpendicular to that space. Orthogonality to every column produces the normal equations, so the calculation is the algebraic form of projection.
How are orthogonal diagonalisation and the SVD related but different?
Orthogonal diagonalisation is a special structure for real symmetric square matrices using eigenvectors. The SVD applies to every real matrix and can use different orthonormal directions in the input and output spaces, joined by singular-value scaling.
Exam move
Use the varied 10-page chapter as a dependency map: define each object, reproduce the fresh worked example, explain both diagrams, answer the two retrieval prompts, and perform an independent check. For least squares, orthogonal matrices, symmetry and svd, revisit the first line where your representation, dimensions, or theorem conditions diverge.
Draw one geometry chain from target to projection to orthogonal residual, then translate each arrow into a matrix equation. Audit orthogonal matrices by checking column inner products and connect symmetry to an orthonormal eigenbasis. For the SVD, label input directions, scaling, and output directions and explain how the decomposition maps the unit sphere.
Working through Least Squares, Orthogonal Matrices, Symmetry and SVD in MAST10007? Sia is AskSia’s AI Mathematics tutor — ask any MAST10007 Least Squares, Orthogonal Matrices, Symmetry and SVD question and get a clear, step-by-step explanation grounded in how MAST10007 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
