Improving Regression Performance with Distributional Losses
arXiv:1806.04613
Abstract
There is growing evidence that converting targets to soft targets in supervised learning can provide considerable gains in performance. Much of this work has considered classification, converting hard zero-one values to soft labels---such as by adding label noise, incorporating label ambiguity or using distillation. In parallel, there is some evidence from a regression setting in reinforcement learning that learning distributions can improve performance. In this work, we investigate the reasons for this improvement, in a regression setting. We introduce a novel distributional regression loss, and similarly find it significantly improves prediction accuracy. We investigate several common hypotheses, around reducing overfitting and improved representations. We instead find evidence for an alternative hypothesis: this loss is easier to optimize, with better behaved gradients, resulting in improved generalization. We provide theoretical support for this alternative hypothesis, by characterizing the norm of the gradients of this loss.
arXiv comment12 pages, 4 figures. To appear in Proceedings of the 35th International Conference on Machine Learning, 2018
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- Scikit-learn: Machine Learning in Python
- Distilling the Knowledge in a Neural Network
- Pixel Recurrent Neural Networks
- Train faster, generalize better: Stability of stochastic gradient descent
- A Distributional Perspective on Reinforcement Learning
- Robust Loss Functions under Label Noise for Deep Neural Networks
- Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
- Reward Augmented Maximum Likelihood for Neural Structured Prediction
- Learning in an Uncertain World: Representing Ambiguity Through Multiple Hypotheses