Why to "grow" and "harvest" deep learning models?
arXiv:2008.03501
Abstract
Current expectations from training deep learning models with gradient-based methods include: 1) transparency; 2) high convergence rates; 3) high inductive biases. While the state-of-art methods with adaptive learning rate schedules are fast, they still fail to meet the other two requirements. We suggest reconsidering neural network models in terms of single-species population dynamics where adaptation comes naturally from open-ended processes of "growth" and "harvesting". We show that the stochastic gradient descent (SGD) with two balanced pre-defined values of per capita growth and harvesting rates outperform the most common adaptive gradient methods in all of the three requirements.
References in corpus (7)
- ADADELTA: An Adaptive Learning Rate Method
- An overview of gradient descent optimization algorithms
- The Marginal Value of Adaptive Gradient Methods in Machine Learning
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate
- Meta-Learning by the Baldwin Effect