High Dimensional Classification via Regularized and Unregularized Empirical Risk Minimization: Precise Error and Optimal Loss

Abstract

This article provides, through theoretical analysis, an in-depth understanding of the classification performance of the empirical risk minimization framework, in both ridge-regularized and unregularized cases, when high dimensional data are considered. Focusing on the fundamental problem of separating a two-class Gaussian mixture, the proposed analysis allows for a precise prediction of the classification error for a set of numerous data vectors $\mathbf{x}\in {\mathbb{R}}^p$ of sufficiently large dimension $p$. This precise error depends on the loss function, the number of training samples, and the statistics of the mixture data model. It is shown to hold beyond Gaussian distribution under some additional non-sparsity condition of the data statistics. Building upon this quantitative error analysis, we identify the simple square loss as the optimal choice for high dimensional classification in both ridge-regularized and unregularized cases, regardless of the number of training samples.

References in corpus (6)

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A modern maximum-likelihood theory for high-dimensional logistic regression
• The generalization error of random features regression: Precise asymptotics and double descent curve
• A Large Dimensional Analysis of Least Squares Support Vector Machines
• The Dynamics of Learning: A Random Matrix Approach
• Sharp Asymptotics and Optimal Performance for Inference in Binary Models

Cited by in corpus (5)

• Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed
• The Quantum Path Kernel: a Generalized Quantum Neural Tangent Kernel for Deep Quantum Machine Learning
• Wide flat minima and optimal generalization in classifying high-dimensional Gaussian mixtures
• Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions
• On the Regularization of Autoencoders