Singular Value Decomposition

SVD

For any $m\times n$ real matrix $A$, we can find $mm\times m$ real orthogonal matrix $U$ and $n\times n$ real orthogonal matrix $V$ s.t. $A=U\Sigma V^T$, where $\Sigma =\left[\begin{array}{cc}D& O\\ O& O\end{array}\right]$ and $D$ is a real $r\times r$ diagonal matrix with positive diagonal entries, and $r$ is the rank of $A$.

• If the diagonal entries of $\Sigma$ are ${\sigma }_1,\dots ,{\sigma }_r$, then $A=\Sigma {\sigma }_i{\mathbf{u}}_i{\mathbf{v}}_i^T$.
• $A^{T}A=V{\Sigma }^T\Sigma V^T$. This is the spectral decomposition for $A^{T}A$.

A singular value $\sigma >0$ for $A$ is the square root of a non-zero eigenvalue of $A^{T}A$ or $A{A}^T$.

Because $A=\Sigma {\sigma }_i{\mathbf{u}}_i{\mathbf{v}}_i^T$, $A{\mathbf{v}}_i={\sigma }_i{\mathbf{u}}_i$.

For a right singular vector $\mathbf{v}$ for singular valur $\sigma$, $\Vert A\mathbf{v}\Vert =\sigma \Vert \mathbf{v}\Vert$.

• ${\mathbf{u}}_1,\dots ,{\mathbf{u}}_r$ are all in $\text{Ran}(A)$, ${\mathbf{u}}_{r+1},\dots ,{\mathbf{u}}_m$ are all in $\text{Ker}(A^T)$. ${\mathbf{v}}_1,\dots ,{\mathbf{v}}_r$ are all in $\text{Ran}(A^T)$, ${\mathbf{v}}_{r+1},\dots ,{\mathbf{v}}_n$ are all in $\text{Ker}(A)$. SVD means simultaneously finding the best orthonormal basis for all subspaces related to $A$.
• Low Rank Approximation: Use the sum of the largest $k$ rank one components of SVD instead of $r$ components. $A\approx {\sum }_{i=1}^k{\sigma }_i{\mathbf{u}}_i{\mathbf{v}}_i^T$.