Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution

Abstract

Neural Architecture Search aims at automatically finding neural architectures that are competitive with architectures designed by human experts. While recent approaches have achieved state-of-the-art predictive performance for image recognition, they are problematic under resource constraints for two reasons: (1)the neural architectures found are solely optimized for high predictive performance, without penalizing excessive resource consumption, (2) most architecture search methods require vast computational resources. We address the first shortcoming by proposing LEMONADE, an evolutionary algorithm for multi-objective architecture search that allows approximating the entire Pareto-front of architectures under multiple objectives, such as predictive performance and number of parameters, in a single run of the method. We address the second shortcoming by proposing a Lamarckian inheritance mechanism for LEMONADE which generates children networks that are warmstarted with the predictive performance of their trained parents. This is accomplished by using (approximate) network morphism operators for generating children. The combination of these two contributions allows finding models that are on par or even outperform both hand-crafted as well as automatically-designed networks.

Published as a conference paper at ICLR, International Conference on Learning Representations, 2019

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• mixup: Beyond Empirical Risk Minimization
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• DARTS: Differentiable Architecture Search
• Designing Neural Network Architectures using Reinforcement Learning
• Efficient Architecture Search by Network Transformation
• Shake-Shake regularization
• Progressive Neural Architecture Search
• Path-Level Network Transformation for Efficient Architecture Search

Cited by in corpus (7)

• Automated Machine Learning: State-of-The-Art and Open Challenges
• Towards Green Automated Machine Learning: Status Quo and Future Directions
• Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
• Trainless Model Performance Estimation for Neural Architecture Search
• Neural Architecture Search via Bregman Iterations
• Assured Neural Network Architectures for Control and Identification of Nonlinear Systems
• Neural Architecture Search in operational context: a remote sensing case-study