A Model of Double Descent for High-dimensional Binary Linear Classification

Abstract

We consider a model for logistic regression where only a subset of features of size $p$ is used for training a linear classifier over $n$ training samples. The classifier is obtained by running gradient descent (GD) on logistic loss. For this model, we investigate the dependence of the classification error on the overparameterization ratio $\kappa =p/n$. First, building on known deterministic results on the implicit bias of GD, we uncover a phase-transition phenomenon for the case of Gaussian features: the classification error of GD is the same as that of the maximum-likelihood (ML) solution when $\kappa <{\kappa }_{\star }$, and that of the max-margin (SVM) solution when $\kappa >{\kappa }_{\star }$. Next, using the convex Gaussian min-max theorem (CGMT), we sharply characterize the performance of both the ML and the SVM solutions. Combining these results, we obtain curves that explicitly characterize the classification error for varying values of $\kappa$. The numerical results validate the theoretical predictions and unveil double-descent phenomena that complement similar recent findings in linear regression settings as well as empirical observations in more complex learning scenarios.

arXiv commentShort version submitted to ICASSP 2020; Updates in 2nd version: revised proofs, typos fixed, extended discussions and numerical illustrations

References in corpus (12)

• Benign Overfitting in Linear Regression
• A modern maximum-likelihood theory for high-dimensional logistic regression
• Scaling description of generalization with number of parameters in deep learning
• Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
• A framework to characterize performance of LASSO algorithms
• The LASSO with Non-linear Measurements is Equivalent to One With Linear Measurements
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• Does data interpolation contradict statistical optimality?
• Symbol Error Rate Performance of Box-relaxation Decoders in Massive MIMO
• Margins, Shrinkage, and Boosting
• Phase Retrieval via Polytope Optimization: Geometry, Phase Transitions, and New Algorithms
• Regularization in High-Dimensional Regression and Classification via Random Matrix Theory

Cited by in corpus (3)

• A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-${\ell }_1$-Norm Interpolated Classifiers
• Wide flat minima and optimal generalization in classifying high-dimensional Gaussian mixtures
• Asymptotic Characterisation of Regularised Zero-Forcing Receiver for Imperfect and Correlated Massive MIMO Systems with Optimal Power Allocation