Sufficient dimension reduction

In statistics, sufficient dimension reduction (SDR) is a paradigm for analyzing data that combines the ideas of dimension reduction with the concept of sufficiency.

Dimension reduction has long been a primary goal of regression analysis. Given a response variable y and a p-dimensional predictor vector , regression analysis aims to study the distribution of , the conditional distribution of given . A dimension reduction is a function that maps to a subset of , k < p, thereby reducing the dimension of .[1] For example, may be one or more linear combinations of .

A dimension reduction is said to be sufficient if the distribution of is the same as that of . In other words, no information about the regression is lost in reducing the dimension of if the reduction is sufficient.[1]

Graphical motivation

edit

In a regression setting, it is often useful to summarize the distribution of   graphically. For instance, one may consider a scatterplot of   versus one or more of the predictors or a linear combination of the predictors. A scatterplot that contains all available regression information is called a sufficient summary plot.

When   is high-dimensional, particularly when  , it becomes increasingly challenging to construct and visually interpret sufficiency summary plots without reducing the data. Even three-dimensional scatter plots must be viewed via a computer program, and the third dimension can only be visualized by rotating the coordinate axes. However, if there exists a sufficient dimension reduction   with small enough dimension, a sufficient summary plot of   versus   may be constructed and visually interpreted with relative ease.

Hence sufficient dimension reduction allows for graphical intuition about the distribution of  , which might not have otherwise been available for high-dimensional data.

Most graphical methodology focuses primarily on dimension reduction involving linear combinations of  . The rest of this article deals only with such reductions.

Dimension reduction subspace

edit

Suppose   is a sufficient dimension reduction, where   is a   matrix with rank  . Then the regression information for   can be inferred by studying the distribution of  , and the plot of   versus   is a sufficient summary plot.

Without loss of generality, only the space spanned by the columns of   need be considered. Let   be a basis for the column space of  , and let the space spanned by   be denoted by  . It follows from the definition of a sufficient dimension reduction that

 

where   denotes the appropriate distribution function. Another way to express this property is

 

or   is conditionally independent of  , given  . Then the subspace   is defined to be a dimension reduction subspace (DRS).[2]

Structural dimensionality

edit

For a regression  , the structural dimension,  , is the smallest number of distinct linear combinations of   necessary to preserve the conditional distribution of  . In other words, the smallest dimension reduction that is still sufficient maps   to a subset of  . The corresponding DRS will be d-dimensional.[2]

Minimum dimension reduction subspace

edit

A subspace   is said to be a minimum DRS for   if it is a DRS and its dimension is less than or equal to that of all other DRSs for  . A minimum DRS   is not necessarily unique, but its dimension is equal to the structural dimension   of  , by definition.[2]

If   has basis   and is a minimum DRS, then a plot of y versus   is a minimal sufficient summary plot, and it is (d + 1)-dimensional.

Central subspace

edit

If a subspace   is a DRS for  , and if   for all other DRSs  , then it is a central dimension reduction subspace, or simply a central subspace, and it is denoted by  . In other words, a central subspace for   exists if and only if the intersection   of all dimension reduction subspaces is also a dimension reduction subspace, and that intersection is the central subspace  .[2]

The central subspace   does not necessarily exist because the intersection   is not necessarily a DRS. However, if   does exist, then it is also the unique minimum dimension reduction subspace.[2]

Existence of the central subspace

edit

While the existence of the central subspace   is not guaranteed in every regression situation, there are some rather broad conditions under which its existence follows directly. For example, consider the following proposition from Cook (1998):

Let   and   be dimension reduction subspaces for  . If   has density   for all   and   everywhere else, where   is convex, then the intersection   is also a dimension reduction subspace.

It follows from this proposition that the central subspace   exists for such  .[2]

Methods for dimension reduction

edit

There are many existing methods for dimension reduction, both graphical and numeric. For example, sliced inverse regression (SIR) and sliced average variance estimation (SAVE) were introduced in the 1990s and continue to be widely used.[3] Although SIR was originally designed to estimate an effective dimension reducing subspace, it is now understood that it estimates only the central subspace, which is generally different.

More recent methods for dimension reduction include likelihood-based sufficient dimension reduction,[4] estimating the central subspace based on the inverse third moment (or kth moment),[5] estimating the central solution space,[6] graphical regression,[2] envelope model, and the principal support vector machine.[7] For more details on these and other methods, consult the statistical literature.

Principal components analysis (PCA) and similar methods for dimension reduction are not based on the sufficiency principle.

Example: linear regression

edit

Consider the regression model

 

Note that the distribution of   is the same as the distribution of  . Hence, the span of   is a dimension reduction subspace. Also,   is 1-dimensional (unless  ), so the structural dimension of this regression is  .

The OLS estimate   of   is consistent, and so the span of   is a consistent estimator of  . The plot of   versus   is a sufficient summary plot for this regression.

See also

edit

Notes

edit
  1. ^ a b Cook & Adragni (2009) Sufficient Dimension Reduction and Prediction in Regression In: Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 367(1906): 4385–4405
  2. ^ a b c d e f g Cook, R.D. (1998) Regression Graphics: Ideas for Studying Regressions Through Graphics, Wiley ISBN 0471193658
  3. ^ Li, K-C. (1991) Sliced Inverse Regression for Dimension Reduction In: Journal of the American Statistical Association, 86(414): 316–327
  4. ^ Cook, R.D. and Forzani, L. (2009) "Likelihood-Based Sufficient Dimension Reduction", Journal of the American Statistical Association, 104(485): 197–208
  5. ^ Yin, X. and Cook, R.D. (2003) Estimating Central Subspaces via Inverse Third Moments In: Biometrika, 90(1): 113–125
  6. ^ Li, B. and Dong, Y.D. (2009) Dimension Reduction for Nonelliptically Distributed Predictors In: Annals of Statistics, 37(3): 1272–1298
  7. ^ Li, Bing; Artemiou, Andreas; Li, Lexin (2011). "Principal support vector machines for linear and nonlinear sufficient dimension reduction". The Annals of Statistics. 39 (6): 3182–3210. arXiv:1203.2790. doi:10.1214/11-AOS932. S2CID 88519106.

References

edit
edit