User:Scut723/sandbox

The singular value decomposition (SVD) is one of the most powerful tools in theoretical and numerical linear algebra. The utility comes from three basic properties:

• Every matrix has an SVD.
• The SVD provides an orthonormal resolution for the four invariant subspaces.
• The SVD provides an ordered list of singular values.

The Singular Value Decomposition Theorem

The singular value decomposition is the most powerful - and most expensive - decomposition tool in linear algebra. The power comes from the resolution of the four fundamental subspaces as well as the eigenvalues.

Existence

Every matrix has a singular value decomposition. Given a matrix $\mathbf {A} \in \mathbb {C} _{\rho }^{m\times n}$, that is, with $m$ rows, $n$ columns, and rank $\rho$, the SVD can be written as

$\mathbf {A} =\mathbf {U} \Sigma \mathbf {V} ^{*}$,

where

• $\mathbf {U} \in \mathbb {C} ^{m\times m}$ resolves the column space,
• $\mathbf {V} \in \mathbb {C} ^{n\times n}$ resolves the row space,
• $\mathbf {\Sigma } \in \mathbb {R} _{\rho }^{m\times n}$ contains the singular values.

The domain matrices are unitary:

$\mathbf {U} \mathbf {U} ^{*}=\mathbf {U} ^{*}\mathbf {U} =\mathbf {I} _{m}$

$\mathbf {V} \mathbf {V} ^{*}=\mathbf {V} ^{*}\mathbf {V} =\mathbf {I} _{n}$

Uniqueness

The singular values are unique, therefore the matrices $\mathbf {S}$ and $\Sigma$ are unique. Typically the domain matrices are not unique. For example, there could be two different decompositions such that

$\mathbf {A} =\mathbf {U} _{1}\mathbf {\Sigma } \mathbf {V} _{1}^{*}=\mathbf {U} _{2}\mathbf {\Sigma } \mathbf {V} _{2}^{*}$

Subspace decomposition

Fundamental Theorem of Linear Algebra

The [Fundamental Theorem of Linear Algebra] states that a matrix $\mathbf {A} \in \mathbb {C} _{\rho }^{m\times n}$ induces a row space (or domain) $\mathbb {C} ^{n}$ and a column space (or codomain) $\mathbb {C} ^{m}$. The row space and the column space each have an orthogonal decomposition into a range space and a null space:

• $\mathbb {C} ^{n}$ = ${\color {Blue}{\overline {{\mathcal {R}}\left(\mathbf {A} ^{*}\right)}}}$ $\oplus$ ${\color {Red}{{\mathcal {N}}\left(\mathbf {A} \right)}}$ (domain),

• $\mathbb {C} ^{m}$ = ${\color {Blue}{\overline {{\mathcal {R}}\left(\mathbf {A} \right)}}}$ $\oplus$ ${\color {Red}{{\mathcal {N}}\left(\mathbf {A} ^{*}\right)}}$ (codomain),

where the overbear represents the set closure required in infinite dimensional spaces.

${\color {Blue}{x^{2}}}+{\color {Orange}{2x}}-{\color {LimeGreen}{1}}$

Block structure

$x^{2}+y^{2}+z^{2}=1$ (1)

${\begin{pmatrix}x&y\\z&v\end{pmatrix}}$

Casting the SVD in block structure emphasizes its subspace decomposition;

$\mathbf {A} =\mathbf {U} \Sigma \mathbf {V} ^{*}=\left[{\begin{array}{cc}{\color {Blue}{\mathbf {U} _{\mathcal {R}}}}&{\color {Red}{\mathbf {U} _{\mathcal {N}}}}\\\end{array}}\right]\left[{\begin{array}{c|c}\mathbf {S} &\mathbf {0} \\\hline \mathbf {0} &\mathbf {0} \end{array}}\right]\left[{\begin{array}{c}{\color {Blue}{\mathbf {V} _{\mathcal {R}}^{*}}}\\{\color {Red}{\mathbf {V} _{\mathcal {N}}^{*}}}\end{array}}\right]$

Geometry of the SVD

The mapping action of a matrix demonstrates the geometry of the SVD. A matrix is an operator which maps an $n-$vector into an $m-$vector

File:/Users/rditldmt/Dropbox/Wiki/svd/movies/AS2.mov

The mapping action of a matrix

$\mathbf {A} \colon \mathbb {C} ^{n}\mapsto \mathbb {C} ^{m}$

Analytic computation

Matrix action on unit circle.

Examples

Full row and column rank

$m=n=\rho =2$

$\left[{\begin{array}{cr}1&-1\\2&2\\\end{array}}\right]$

Matrix action on unit circle.

Matrix action on unit circle.

Matrix action on unit circle.

Orange	Apple
Bread	Pie
Butter	Ice cream