Regression as Classification: Influence of Task Formulation on Neural Network Features
arXiv:2211.05641
Abstract
Neural networks can be trained to solve regression problems by using gradient-based methods to minimize the square loss. However, practitioners often prefer to reformulate regression as a classification problem, observing that training on the cross entropy loss results in better performance. By focusing on two-layer ReLU networks, which can be fully characterized by measures over their feature space, we explore how the implicit bias induced by gradient-based optimization could partly explain the above phenomenon. We provide theoretical evidence that the regression formulation yields a measure whose support can differ greatly from that for classification, in the case of one-dimensional data. Our proposed optimal supports correspond directly to the features learned by the input layer of the network. The different nature of these supports sheds light on possible optimization difficulties the square loss could encounter during training, and we present empirical results illustrating this phenomenon.
References in corpus (10)
- Neural Machine Translation by Jointly Learning to Align and Translate
- Sequence to Sequence Learning with Neural Networks
- DeepPose: Human Pose Estimation via Deep Neural Networks
- Solving Rubik's Cube with a Robot Hand
- Breaking the Curse of Dimensionality with Convex Neural Networks
- Qualitatively characterizing neural network optimization problems
- In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
- Generative Image Modeling Using Spatial LSTMs
- On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
- Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss