Sub-Architecture Ensemble Pruning in Neural Architecture Search

Abstract

Neural architecture search (NAS) is gaining more and more attention in recent years due to its flexibility and remarkable capability to reduce the burden of neural network design. To achieve better performance, however, the searching process usually costs massive computations that might not be affordable for researchers and practitioners. While recent attempts have employed ensemble learning methods to mitigate the enormous computational cost, however, they neglect a key property of ensemble methods, namely diversity, which leads to collecting more similar sub-architectures with potential redundancy in the final design. To tackle this problem, we propose a pruning method for NAS ensembles called "Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP)." It targets to leverage diversity and to achieve sub-ensemble architectures at a smaller size with comparable performance to ensemble architectures that are not pruned. Three possible solutions are proposed to decide which sub-architectures to prune during the searching process. Experimental results exhibit the effectiveness of the proposed method by largely reducing the number of sub-architectures without degrading the performance.

Accepted by TNNLS. This work was done when the first author was a visiting research scholar at Texas A&M University

References in corpus (14)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Neural Architecture Search: A Survey
• DARTS: Differentiable Architecture Search
• Designing Neural Network Architectures using Reinforcement Learning
• Efficient Architecture Search by Network Transformation
• A Survey on Neural Architecture Search
• Path-Level Network Transformation for Efficient Architecture Search
• When does Diversity Help Generalization in Classification Ensembles?
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Ensemble Pruning based on Objection Maximization with a General Distributed Framework
• Differentiable Architecture Search with Ensemble Gumbel-Softmax
• NADS: Neural Architecture Distribution Search for Uncertainty Awareness
• Improving Neural Architecture Search Image Classifiers via Ensemble Learning

Cited by in corpus (1)

• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift