Splitting Steepest Descent for Growing Neural Architectures

Abstract

We develop a progressive training approach for neural networks which adaptively grows the network structure by splitting existing neurons to multiple off-springs. By leveraging a functional steepest descent idea, we derive a simple criterion for deciding the best subset of neurons to split and a splitting gradient for optimally updating the off-springs. Theoretically, our splitting strategy is a second-order functional steepest descent for escaping saddle points in an $\infty$-Wasserstein metric space, on which the standard parametric gradient descent is a first-order steepest descent. Our method provides a new computationally efficient approach for optimizing neural network structures, especially for learning lightweight neural architectures in resource-constrained settings.

arXiv comment33rd Conference on Neural Information Processing Systems (NeurIPS 2019)

References in corpus (9)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• DARTS: Differentiable Architecture Search
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Rethinking the Value of Network Pruning
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Hello Edge: Keyword Spotting on Microcontrollers
• SNAS: Stochastic Neural Architecture Search
• Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions
• On the Equivalence between Herding and Conditional Gradient Algorithms

Cited by in corpus (4)

• PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splitting
• BinaryBERT: Pushing the Limit of BERT Quantization
• A Differential Game Theoretic Neural Optimizer for Training Residual Networks
• Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices