Fitting very flexible models: Linear regression with large numbers of parameters

Abstract

There are many uses for linear fitting; the context here is interpolation and denoising of data, as when you have calibration data and you want to fit a smooth, flexible function to those data. Or you want to fit a flexible function to de-trend a time series or normalize a spectrum. In these contexts, investigators often choose a polynomial basis, or a Fourier basis, or wavelets, or something equally general. They also choose an order, or number of basis functions to fit, and (often) some kind of regularization. We discuss how this basis-function fitting is done, with ordinary least squares and extensions thereof. We emphasize that it is often valuable to choose far more parameters than data points, despite folk rules to the contrary: Suitably regularized models with enormous numbers of parameters generalize well and make good predictions for held-out data; over-fitting is not (mainly) a problem of having too many parameters. It is even possible to take the limit of infinite parameters, at which, if the basis and regularization are chosen correctly, the least-squares fit becomes the mean of a Gaussian process. We recommend cross-validation as a good empirical method for model selection (for example, setting the number of parameters and the form of the regularization), and jackknife resampling as a good empirical method for estimating the uncertainties of the predictions made by the model. We also give advice for building stable computational implementations.

arXiv commentall code used to make the figures is available at https://github.com/davidwhogg/FlexibleLinearModels

References in corpus (9)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Fast and scalable Gaussian process modeling with applications to astronomical time series
• Benign Overfitting in Linear Regression
• The Cannon: A data-driven approach to stellar label determination
• Wiener Reconstruction of The Large Scale Structure
• K2SC: Flexible systematics correction and detrending of K2 light curves using Gaussian Process regression
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• A jamming transition from under- to over-parametrization affects loss landscape and generalization
• The sample complexity of weighted sparse approximation

Cited by in corpus (4)

• StarHorse results for spectroscopic surveys + Gaia DR3: Chrono-chemical populations in the solar vicinity, the genuine thick disk, and young-alpha rich stars
• Half-sibling regression meets exoplanet imaging: PSF modeling and subtraction using a flexible, domain knowledge-driven, causal framework
• Detecting the periodicity of highly irregularly sampled light-curves with Gaussian processes: the case of SDSSJ025214.67-002813.7
• exoALMA. VIII. Probabilistic Moment Maps and Data Products using Non-parametric Linear Models