Neural Architecture Search via Bregman Iterations
arXiv:2106.02479
Abstract
We propose a novel strategy for Neural Architecture Search (NAS) based on Bregman iterations. Starting from a sparse neural network our gradient-based one-shot algorithm gradually adds relevant parameters in an inverse scale space manner. This allows the network to choose the best architecture in the search space which makes it well-designed for a given task, e.g., by adding neurons or skip connections. We demonstrate that using our approach one can unveil, for instance, residual autoencoders for denoising, deblurring, and classification tasks. Code is available at https://github.com/TimRoith/BregmanLearning.
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- An overview of gradient descent optimization algorithms
- Neural Architecture Search with Reinforcement Learning
- DARTS: Differentiable Architecture Search
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- To prune, or not to prune: exploring the efficacy of pruning for model compression
- Neural Ordinary Differential Equations
- Group Sparse Regularization for Deep Neural Networks
- Sparse Networks from Scratch: Faster Training without Losing Performance
- BayesNAS: A Bayesian Approach for Neural Architecture Search
- Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution
- Momentum Residual Neural Networks
- A Bregman Learning Framework for Sparse Neural Networks
- Neural Architecture Search as Sparse Supernet