Learnable Embedding Space for Efficient Neural Architecture Compression

Abstract

We propose a method to incrementally learn an embedding space over the domain of network architectures, to enable the careful selection of architectures for evaluation during compressed architecture search. Given a teacher network, we search for a compressed network architecture by using Bayesian Optimization (BO) with a kernel function defined over our proposed embedding space to select architectures for evaluation. We demonstrate that our search algorithm can significantly outperform various baseline methods, such as random search and reinforcement learning (Ashok et al., 2018). The compressed architectures found by our method are also better than the state-of-the-art manually-designed compact architecture ShuffleNet (Zhang et al., 2018). We also demonstrate that the learned embedding space can be transferred to new settings for architecture search, such as a larger teacher network or a teacher network in a different architecture family, without any training. Code is publicly available here: https://github.com/Friedrich1006/ESNAC .

ICLR 2019 - Code available here: https://github.com/Friedrich1006/ESNAC

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Neural Architecture Search: A Survey
• Efficient Neural Architecture Search via Parameter Sharing
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Hierarchical Representations for Efficient Architecture Search
• Progressive Neural Architecture Search
• MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
• Predictive Entropy Search for Multi-objective Bayesian Optimization
• Stochastic Variational Deep Kernel Learning
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces
• DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures

Cited by in corpus (9)

• Neural Architecture Search: A Survey
• Random Search and Reproducibility for Neural Architecture Search
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Population-Based Black-Box Optimization for Biological Sequence Design
• Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints
• Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models
• One-Shot Neural Architecture Search via Compressive Sensing
• Neighborhood-Aware Neural Architecture Search
• Zero-shot task adaptation by homoiconic meta-mapping