Nonparametric Neural Networks
arXiv:1712.05440
Abstract
Automatically determining the optimal size of a neural network for a given task without prior information currently requires an expensive global search and training many networks from scratch. In this paper, we address the problem of automatically finding a good network size during a single training cycle. We introduce *nonparametric neural networks*, a non-probabilistic framework for conducting optimization over all possible network sizes and prove its soundness when network growth is limited via an L_p penalty. We train networks under this framework by continuously adding new units while eliminating redundant units via an L_2 penalty. We employ a novel optimization algorithm, which we term *adaptive radial-angular gradient descent* or *AdaRad*, and obtain promising results.
ICLR 2017
References in corpus (7)
- Distilling the Knowledge in a Neural Network
- ADADELTA: An Adaptive Learning Rate Method
- Neural Architecture Search with Reinforcement Learning
- Do Deep Nets Really Need to be Deep?
- Gradient-based Hyperparameter Optimization through Reversible Learning
- DSD: Dense-Sparse-Dense Training for Deep Neural Networks
- Steps Toward Deep Kernel Methods from Infinite Neural Networks