Neural Architecture Search via Bregman Iterations

Abstract

We propose a novel strategy for Neural Architecture Search (NAS) based on Bregman iterations. Starting from a sparse neural network our gradient-based one-shot algorithm gradually adds relevant parameters in an inverse scale space manner. This allows the network to choose the best architecture in the search space which makes it well-designed for a given task, e.g., by adding neurons or skip connections. We demonstrate that using our approach one can unveil, for instance, residual autoencoders for denoising, deblurring, and classification tasks. Code is available at https://github.com/TimRoith/BregmanLearning.

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Architecture Search with Reinforcement Learning
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Group Sparse Regularization for Deep Neural Networks
• BayesNAS: A Bayesian Approach for Neural Architecture Search