Model Selection Techniques -- An Overview

Abstract

In the era of big data, analysts usually explore various statistical models or machine learning methods for observed data in order to facilitate scientific discoveries or gain predictive power. Whatever data and fitting procedures are employed, a crucial step is to select the most appropriate model or method from a set of candidates. Model selection is a key ingredient in data analysis for reliable and reproducible statistical inference or prediction, and thus central to scientific studies in fields such as ecology, economics, engineering, finance, political science, biology, and epidemiology. There has been a long history of model selection techniques that arise from researches in statistics, information theory, and signal processing. A considerable number of methods have been proposed, following different philosophies and exhibiting varying performances. The purpose of this article is to bring a comprehensive overview of them, in terms of their motivation, large sample performance, and applicability. We provide integrated and practically relevant discussions on theoretical properties of state-of- the-art model selection approaches. We also share our thoughts on some controversial views on the practice of model selection.

accepted by IEEE SIGNAL PROCESSING MAGAZINE

References in corpus (23)

• Discussion of "Least angle regression" by Efron et al
• Rejoinder to "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Nearly unbiased variable selection under minimax concave penalty
• Simultaneous analysis of Lasso and Dantzig selector
• Valid post-selection inference
• Accurate estimation of influenza epidemics using Google search data via ARGO
• Sparse Estimators and the Oracle Property, or the Return of Hodges' Estimator
• Order selection for same-realization predictions in autoregressive processes
• Rodeo: Sparse, greedy nonparametric regression
• Proper local scoring rules
• Accumulated prediction errors, information criteria and optimal forecasting for autoregressive time series
• Perturbation Analysis of Orthogonal Matching Pursuit
• Parametric or nonparametric? A parametricness index for model selection
• Bayesian model comparison with the Hyvärinen score: computation and consistency
• SLANTS: Sequential Adaptive Nonlinear Modeling of Vector Time Series

Cited by in corpus (45)

• Predictive Multiplicity in Classification
• Model Order Selection in DoA Scenarios via Cross-Entropy based Machine Learning Techniques
• Universal and Automatic Elbow Detection for Learning the Effective Number of Components in Model Selection Problems
• Robust Information Criterion for Model Selection in Sparse High-Dimensional Linear Regression Models
• Assisted Learning: A Framework for Multi-Organization Learning
• Joint Modeling of Received Power, Mean Delay, and Delay Spread for Wideband Radio Channels
• Non-Asymptotic Guarantees for Reliable Identification of Granger Causality via the LASSO
• Interval Privacy: A Framework for Privacy-Preserving Data Collection
• On Statistical Efficiency in Learning
• Goodness-of-fit tests on manifolds
• Non-Bayesian Post-Model-Selection Estimation as Estimation Under Model Misspecification
• Low-Rank Matrix Recovery from Noise via an MDL Framework-based Atomic Norm
• The Rate of Convergence of Variation-Constrained Deep Neural Networks
• Maximum Information Extraction Via Clustering and Minimization of Shannon Entropy
• Assisted Learning for Organizations with Limited Imbalanced Data
• Adaptive Sieving with PPDNA: Generating Solution Paths of Exclusive Lasso Models
• Above 99.9% Fidelity Single-Qubit Gates, Two-Qubit Gates, and Readout in a Single Superconducting Quantum Device
• Regularised Text Logistic Regression: Key Word Detection and Sentiment Classification for Online Reviews
• Inferring phenomenological models of first passage processes
• Fourier Series-Based Approximation of Time-Varying Parameters in Ordinary Differential Equations
• Consistent model selection criteria and goodness-of-fit test for affine causal processes
• Learning the hypotheses space from data through a U-curve algorithm
• Regression model selection via log-likelihood ratio and constrained minimum criterion
• Model Averaging Under Flexible Loss Functions
• Automated Resonance Identification in Nuclear Data Evaluation
• Empirical Bayes factors for common hypothesis tests
• A scale of interpretation for likelihood ratios and Bayes factors
• Model Order Selection with Variational Autoencoding
• From Model Selection to Model Averaging: A Comparison for Nested Linear Models
• A hybrid ensemble method with negative correlation learning for regression
• Testing for a Random Walk Structure in the Frequency Evolution of a Tone in Noise
• It's the Best Only When It Fits You Most: Finding Related Models for Serving Based on Dynamic Locality Sensitive Hashing
• CS-ROMER: A novel compressed sensing framework for Faraday depth reconstruction
• Strong consistent model selection for general causal time series
• Quantum Assemblage Tomography
• ASCII: ASsisted Classification with Ignorance Interchange
• Variable Grouping Based Bayesian Additive Regression Tree
• Model Privacy: A Unified Framework for Understanding Model Stealing Attacks and Defenses
• Model Linkage Selection for Cooperative Learning
• To tree or not to tree? Assessing the impact of smoothing the decision boundaries
• Model family selection for classification using Neural Decision Trees
• A constrained minimum criterion for model selection
• Targeted Cross-Validation
• Occam Factor for Gaussian Models With Unknown Variance Structure
• No Cross-Validation Required: An Analytical Framework for Regularized Mixed-Integer Problems (Extended Version)