The geometry of least squares in the 21st century
arXiv:1309.7837 · doi:10.3150/12-BEJSP15
Abstract
It has been over 200 years since Gauss's and Legendre's famous priority dispute on who discovered the method of least squares. Nevertheless, we argue that the normal equations are still relevant in many facets of modern statistics, particularly in the domain of high-dimensional inference. Even today, we are still learning new things about the law of large numbers, first described in Bernoulli's Ars Conjectandi 300 years ago, as it applies to high dimensional inference. The other insight the normal equations provide is the asymptotic Gaussianity of the least squares estimators. The general form of the Gaussian distribution, Gaussian processes, are another tool used in modern high-dimensional inference. The Gaussian distribution also arises via the central limit theorem in describing weak convergence of the usual least squares estimators. In terms of high-dimensional inference, we are still missing the right notion of weak convergence. In this mostly expository work, we try to describe how both the normal equations and the theory of Gaussian processes, what we refer to as the "geometry of least squares," apply to many questions of current interest.
arXiv commentPublished in at http://dx.doi.org/10.3150/12-BEJSP15 the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm)
References in corpus (24)
- Discussion of "Least angle regression" by Efron et al
- Rejoinder to "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Least Angle Regression
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Simultaneous analysis of Lasso and Dantzig selector
- Message Passing Algorithms for Compressed Sensing
- Pathwise coordinate optimization
- NESTA: A Fast and Accurate First-order Method for Sparse Recovery
- The solution path of the generalized lasso
- High-dimensional variable selection
- Support union recovery in high-dimensional multivariate regression
- Degrees of freedom in lasso problems
- Statistical significance in high-dimensional linear models
- Group Lasso with Overlaps: the Latent Group Lasso approach
- Random fields of multivariate test statistics, with applications to shape analysis
- Validity of the expected Euler characteristic heuristic
- A Gaussian kinematic formula
- High level excursion set geometry for non-Gaussian infinitely divisible random fields