Projection (linear algebra)

(Redirected from Orthogonal projection)

In linear algebra and functional analysis, a projection is a linear transformation from a vector space to itself (an endomorphism) such that . That is, whenever is applied twice to any vector, it gives the same result as if it were applied once (i.e. is idempotent). It leaves its image unchanged.[1] This definition of "projection" formalizes and generalizes the idea of graphical projection. One can also consider the effect of a projection on a geometrical object by examining the effect of the projection on points in the object.

The transformation P is the orthogonal projection onto the line m.

Definitions edit

A projection on a vector space   is a linear operator   such that  .

When   has an inner product and is complete, i.e. when   is a Hilbert space, the concept of orthogonality can be used. A projection   on a Hilbert space   is called an orthogonal projection if it satisfies   for all  . A projection on a Hilbert space that is not orthogonal is called an oblique projection.

Projection matrix edit

  • A square matrix   is called a projection matrix if it is equal to its square, i.e. if  .[2]: p. 38 
  • A square matrix   is called an orthogonal projection matrix if   for a real matrix, and respectively   for a complex matrix, where   denotes the transpose of   and   denotes the adjoint or Hermitian transpose of  .[2]: p. 223 
  • A projection matrix that is not an orthogonal projection matrix is called an oblique projection matrix.

The eigenvalues of a projection matrix must be 0 or 1.

Examples edit

Orthogonal projection edit

For example, the function which maps the point   in three-dimensional space   to the point   is an orthogonal projection onto the xy-plane. This function is represented by the matrix

 

The action of this matrix on an arbitrary vector is

 

To see that   is indeed a projection, i.e.,  , we compute

 

Observing that   shows that the projection is an orthogonal projection.

Oblique projection edit

A simple example of a non-orthogonal (oblique) projection is

 

Via matrix multiplication, one sees that

 
showing that   is indeed a projection.

The projection   is orthogonal if and only if   because only then  

Properties and classification edit

 
The transformation T is the projection along k onto m. The range of T is m and the kernel is k.

Idempotence edit

By definition, a projection   is idempotent (i.e.  ).

Open map edit

Every projection is an open map, meaning that it maps each open set in the domain to an open set in the subspace topology of the image.[citation needed] That is, for any vector   and any ball   (with positive radius) centered on  , there exists a ball   (with positive radius) centered on   that is wholly contained in the image  .

Complementarity of image and kernel edit

Let   be a finite-dimensional vector space and   be a projection on  . Suppose the subspaces   and   are the image and kernel of   respectively. Then   has the following properties:

  1.   is the identity operator   on  :
     
  2. We have a direct sum  . Every vector   may be decomposed uniquely as   with   and  , and where  

The image and kernel of a projection are complementary, as are   and  . The operator   is also a projection as the image and kernel of   become the kernel and image of   and vice versa. We say   is a projection along   onto   (kernel/image) and   is a projection along   onto  .

Spectrum edit

In infinite-dimensional vector spaces, the spectrum of a projection is contained in   as

 
Only 0 or 1 can be an eigenvalue of a projection. This implies that an orthogonal projection   is always a positive semi-definite matrix. In general, the corresponding eigenspaces are (respectively) the kernel and range of the projection. Decomposition of a vector space into direct sums is not unique. Therefore, given a subspace  , there may be many projections whose range (or kernel) is  .

If a projection is nontrivial it has minimal polynomial  , which factors into distinct linear factors, and thus   is diagonalizable.

Product of projections edit

The product of projections is not in general a projection, even if they are orthogonal. If two projections commute then their product is a projection, but the converse is false: the product of two non-commuting projections may be a projection.

If two orthogonal projections commute then their product is an orthogonal projection. If the product of two orthogonal projections is an orthogonal projection, then the two orthogonal projections commute (more generally: two self-adjoint endomorphisms commute if and only if their product is self-adjoint).

Orthogonal projections edit

When the vector space   has an inner product and is complete (is a Hilbert space) the concept of orthogonality can be used. An orthogonal projection is a projection for which the range   and the kernel   are orthogonal subspaces. Thus, for every   and   in  ,  . Equivalently:

 

A projection is orthogonal if and only if it is self-adjoint. Using the self-adjoint and idempotent properties of  , for any   and   in   we have  ,  , and

 
where   is the inner product associated with  . Therefore,   and   are orthogonal projections.[3] The other direction, namely that if   is orthogonal then it is self-adjoint, follows from the implication from   to
 
for every   and   in  ; thus  .

The existence of an orthogonal projection onto a closed subspace follows from the Hilbert projection theorem.

Properties and special cases edit

An orthogonal projection is a bounded operator. This is because for every   in the vector space we have, by the Cauchy–Schwarz inequality:

 
Thus  .

For finite-dimensional complex or real vector spaces, the standard inner product can be substituted for  .

Formulas edit

A simple case occurs when the orthogonal projection is onto a line. If   is a unit vector on the line, then the projection is given by the outer product

 
(If   is complex-valued, the transpose in the above equation is replaced by a Hermitian transpose). This operator leaves u invariant, and it annihilates all vectors orthogonal to  , proving that it is indeed the orthogonal projection onto the line containing u.[4] A simple way to see this is to consider an arbitrary vector   as the sum of a component on the line (i.e. the projected vector we seek) and another perpendicular to it,  . Applying projection, we get
 
by the properties of the dot product of parallel and perpendicular vectors.

This formula can be generalized to orthogonal projections on a subspace of arbitrary dimension. Let   be an orthonormal basis of the subspace  , with the assumption that the integer  , and let   denote the   matrix whose columns are  , i.e.,  . Then the projection is given by:[5]

 
which can be rewritten as
 

The matrix   is the partial isometry that vanishes on the orthogonal complement of  , and   is the isometry that embeds   into the underlying vector space. The range of   is therefore the final space of  . It is also clear that   is the identity operator on  .

The orthonormality condition can also be dropped. If   is a (not necessarily orthonormal) basis with  , and   is the matrix with these vectors as columns, then the projection is:[6][7]

 

The matrix   still embeds   into the underlying vector space but is no longer an isometry in general. The matrix   is a "normalizing factor" that recovers the norm. For example, the rank-1 operator   is not a projection if   After dividing by   we obtain the projection   onto the subspace spanned by  .

In the general case, we can have an arbitrary positive definite matrix   defining an inner product  , and the projection   is given by  . Then

 

When the range space of the projection is generated by a frame (i.e. the number of generators is greater than its dimension), the formula for the projection takes the form:  . Here   stands for the Moore–Penrose pseudoinverse. This is just one of many ways to construct the projection operator.

If   is a non-singular matrix and   (i.e.,   is the null space matrix of  ),[8] the following holds:

 

If the orthogonal condition is enhanced to   with   non-singular, the following holds:

 

All these formulas also hold for complex inner product spaces, provided that the conjugate transpose is used instead of the transpose. Further details on sums of projectors can be found in Banerjee and Roy (2014).[9] Also see Banerjee (2004)[10] for application of sums of projectors in basic spherical trigonometry.

Oblique projections edit

The term oblique projections is sometimes used to refer to non-orthogonal projections. These projections are also used to represent spatial figures in two-dimensional drawings (see oblique projection), though not as frequently as orthogonal projections. Whereas calculating the fitted value of an ordinary least squares regression requires an orthogonal projection, calculating the fitted value of an instrumental variables regression requires an oblique projection.

A projection is defined by its kernel and the basis vectors used to characterize its range (which is a complement of the kernel). When these basis vectors are orthogonal to the kernel, then the projection is an orthogonal projection. When these basis vectors are not orthogonal to the kernel, the projection is an oblique projection, or just a projection.

A matrix representation formula for a nonzero projection operator edit

Let   be a linear operator,   such that   and assume that   is not the zero operator. Let the vectors   form a basis for the range of  , and assemble these vectors in the   matrix  . Therefore the integer  , otherwise   and   is the zero operator. The range and the kernel are complementary spaces, so the kernel has dimension  . It follows that the orthogonal complement of the kernel has dimension  . Let   form a basis for the orthogonal complement of the kernel of the projection, and assemble these vectors in the matrix  . Then the projection   (with the condition  ) is given by

 

This expression generalizes the formula for orthogonal projections given above.[11][12] A standard proof of this expression is the following. For any vector   in the vector space  , we can decompose  , where vector   is in the image of  , and vector   So  , and then   is in the kernel of  , which is the null space of   In other words, the vector   is in the column space of   so   for some   dimension vector   and the vector   satisfies   by the construction of  . Put these conditions together, and we find a vector   so that  . Since matrices   and   are of full rank   by their construction, the  -matrix   is invertible. So the equation   gives the vector   In this way,   for any vector   and hence  .

In the case that   is an orthogonal projection, we can take  , and it follows that  . By using this formula, one can easily check that  . In general, if the vector space is over complex number field, one then uses the Hermitian transpose   and has the formula  . Recall that one can define the Moore–Penrose inverse of the matrix   by   since   has full column rank, so  .

Singular values edit

Note that   is also an oblique projection. The singular values of   and   can be computed by an orthonormal basis of  . Let   be an orthonormal basis of   and let   be the orthogonal complement of  . Denote the singular values of the matrix   by the positive values  . With this, the singular values for   are:[13]

 
and the singular values for   are
 
This implies that the largest singular values of   and   are equal, and thus that the matrix norm of the oblique projections are the same. However, the condition number satisfies the relation  , and is therefore not necessarily equal.

Finding projection with an inner product edit

Let   be a vector space (in this case a plane) spanned by orthogonal vectors  . Let   be a vector. One can define a projection of   onto   as

 
where repeated indices are summed over (Einstein sum notation). The vector   can be written as an orthogonal sum such that  .   is sometimes denoted as  . There is a theorem in linear algebra that states that this   is the smallest distance (the orthogonal distance) from   to   and is commonly used in areas such as machine learning.
 
y is being projected onto the vector space V.

Canonical forms edit

Any projection   on a vector space of dimension   over a field is a diagonalizable matrix, since its minimal polynomial divides  , which splits into distinct linear factors. Thus there exists a basis in which   has the form

 

where   is the rank of  . Here   is the identity matrix of size  ,   is the zero matrix of size  , and   is the direct sum operator. If the vector space is complex and equipped with an inner product, then there is an orthonormal basis in which the matrix of P is[14]

 

where  . The integers   and the real numbers   are uniquely determined. Note that  . The factor   corresponds to the maximal invariant subspace on which   acts as an orthogonal projection (so that P itself is orthogonal if and only if  ) and the  -blocks correspond to the oblique components.

Projections on normed vector spaces edit

When the underlying vector space   is a (not necessarily finite-dimensional) normed vector space, analytic questions, irrelevant in the finite-dimensional case, need to be considered. Assume now   is a Banach space.

Many of the algebraic results discussed above survive the passage to this context. A given direct sum decomposition of   into complementary subspaces still specifies a projection, and vice versa. If   is the direct sum  , then the operator defined by   is still a projection with range   and kernel  . It is also clear that  . Conversely, if   is projection on  , i.e.  , then it is easily verified that  . In other words,   is also a projection. The relation   implies   and   is the direct sum  .

However, in contrast to the finite-dimensional case, projections need not be continuous in general. If a subspace   of   is not closed in the norm topology, then the projection onto   is not continuous. In other words, the range of a continuous projection   must be a closed subspace. Furthermore, the kernel of a continuous projection (in fact, a continuous linear operator in general) is closed. Thus a continuous projection   gives a decomposition of   into two complementary closed subspaces:  .

The converse holds also, with an additional assumption. Suppose   is a closed subspace of  . If there exists a closed subspace   such that X = UV, then the projection   with range   and kernel   is continuous. This follows from the closed graph theorem. Suppose xnx and Pxny. One needs to show that  . Since   is closed and {Pxn} ⊂ U, y lies in  , i.e. Py = y. Also, xnPxn = (IP)xnxy. Because   is closed and {(IP)xn} ⊂ V, we have  , i.e.  , which proves the claim.

The above argument makes use of the assumption that both   and   are closed. In general, given a closed subspace  , there need not exist a complementary closed subspace  , although for Hilbert spaces this can always be done by taking the orthogonal complement. For Banach spaces, a one-dimensional subspace always has a closed complementary subspace. This is an immediate consequence of Hahn–Banach theorem. Let   be the linear span of  . By Hahn–Banach, there exists a bounded linear functional   such that φ(u) = 1. The operator   satisfies  , i.e. it is a projection. Boundedness of   implies continuity of   and therefore   is a closed complementary subspace of  .

Applications and further considerations edit

Projections (orthogonal and otherwise) play a major role in algorithms for certain linear algebra problems:

As stated above, projections are a special case of idempotents. Analytically, orthogonal projections are non-commutative generalizations of characteristic functions. Idempotents are used in classifying, for instance, semisimple algebras, while measure theory begins with considering characteristic functions of measurable sets. Therefore, as one can imagine, projections are very often encountered in the context of operator algebras. In particular, a von Neumann algebra is generated by its complete lattice of projections.

Generalizations edit

More generally, given a map between normed vector spaces   one can analogously ask for this map to be an isometry on the orthogonal complement of the kernel: that   be an isometry (compare Partial isometry); in particular it must be onto. The case of an orthogonal projection is when W is a subspace of V. In Riemannian geometry, this is used in the definition of a Riemannian submersion.

See also edit

Notes edit

  1. ^ Meyer, pp 386+387
  2. ^ a b Horn, Roger A.; Johnson, Charles R. (2013). Matrix Analysis, second edition. Cambridge University Press. ISBN 9780521839402.
  3. ^ Meyer, p. 433
  4. ^ Meyer, p. 431
  5. ^ Meyer, equation (5.13.4)
  6. ^ Banerjee, Sudipto; Roy, Anindya (2014), Linear Algebra and Matrix Analysis for Statistics, Texts in Statistical Science (1st ed.), Chapman and Hall/CRC, ISBN 978-1420095388
  7. ^ Meyer, equation (5.13.3)
  8. ^ See also Linear least squares (mathematics) § Properties of the least-squares estimators.
  9. ^ Banerjee, Sudipto; Roy, Anindya (2014), Linear Algebra and Matrix Analysis for Statistics, Texts in Statistical Science (1st ed.), Chapman and Hall/CRC, ISBN 978-1420095388
  10. ^ Banerjee, Sudipto (2004), "Revisiting Spherical Trigonometry with Orthogonal Projectors", The College Mathematics Journal, 35 (5): 375–381, doi:10.1080/07468342.2004.11922099, S2CID 122277398
  11. ^ Banerjee, Sudipto; Roy, Anindya (2014), Linear Algebra and Matrix Analysis for Statistics, Texts in Statistical Science (1st ed.), Chapman and Hall/CRC, ISBN 978-1420095388
  12. ^ Meyer, equation (7.10.39)
  13. ^ Brust, J. J.; Marcia, R. F.; Petra, C. G. (2020), "Computationally Efficient Decompositions of Oblique Projection Matrices", SIAM Journal on Matrix Analysis and Applications, 41 (2): 852–870, doi:10.1137/19M1288115, OSTI 1680061, S2CID 219921214
  14. ^ Doković, D. Ž. (August 1991). "Unitary similarity of projectors". Aequationes Mathematicae. 42 (1): 220–224. doi:10.1007/BF01818492. S2CID 122704926.

References edit

  • Banerjee, Sudipto; Roy, Anindya (2014), Linear Algebra and Matrix Analysis for Statistics, Texts in Statistical Science (1st ed.), Chapman and Hall/CRC, ISBN 978-1420095388
  • Dunford, N.; Schwartz, J. T. (1958). Linear Operators, Part I: General Theory. Interscience.
  • Meyer, Carl D. (2000). Matrix Analysis and Applied Linear Algebra. Society for Industrial and Applied Mathematics. ISBN 978-0-89871-454-8.

External links edit