On the interplay between data structure and loss function in classification problems
arXiv:2103.05524
Abstract
One of the central puzzles in modern machine learning is the ability of heavily overparametrized models to generalize well. Although the low-dimensional structure of typical datasets is key to this behavior, most theoretical studies of overparametrization focus on isotropic inputs. In this work, we instead consider an analytically tractable model of structured data, where the input covariance is built from independent blocks allowing us to tune the saliency of low-dimensional structures and their alignment with respect to the target function. Using methods from statistical physics, we derive a precise asymptotic expression for the train and test error achieved by random feature models trained to classify such data, which is valid for any convex loss function. We study in detail how the data structure affects the double descent curve, and show that in the over-parametrized regime, its impact is greater for logistic loss than for mean-squared loss: the easier the task, the wider the gap in performance at the advantage of the logistic loss. Our insights are confirmed by numerical experiments on MNIST and CIFAR10.
References in corpus (25)
- Sequence to Sequence Learning with Neural Networks
- Language Models are Few-Shot Learners
- Wide Residual Networks
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- On Lazy Training in Differentiable Programming
- The generalization error of random features regression: Precise asymptotics and double descent curve
- Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
- Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
- Scaling description of generalization with number of parameters in deep learning
- A Landscape Analysis of Constraint Satisfaction Problems
- Intrinsic dimension of data representations in deep neural networks
- The simplest model of jamming
- The jamming transition as a paradigm to understand the loss landscape of deep neural networks
- The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
- Generalisation error in learning with random features and the hidden manifold model
- Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks
- Optimal Regularization Can Mitigate Double Descent
- The Gaussian equivalence of generative models for learning with shallow neural networks
- Classification vs regression in overparameterized regimes: Does the loss function matter?
- Multiple Descent: Design Your Own Generalization Curve
- The Intrinsic Dimension of Images and Its Impact on Learning
- Universality Laws for High-Dimensional Learning with Random Features
- Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization
- Asymptotics of Ridge (less) Regression under General Source Condition
- A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent