Apollo: Transferable Architecture Exploration

Abstract

The looming end of Moore's Law and ascending use of deep learning drives the design of custom accelerators that are optimized for specific neural architectures. Architecture exploration for such accelerators forms a challenging constrained optimization problem over a complex, high-dimensional, and structured input space with a costly to evaluate objective function. Existing approaches for accelerator design are sample-inefficient and do not transfer knowledge between related optimizations tasks with different design constraints, such as area and/or latency budget, or neural architecture configurations. In this work, we propose a transferable architecture exploration framework, dubbed Apollo, that leverages recent advances in black-box function optimization for sample-efficient accelerator design. We use this framework to optimize accelerator configurations of a diverse set of neural architectures with alternative design constraints. We show that our framework finds high reward design configurations (up to 24.6% speedup) more sample-efficiently than a baseline black-box optimization approach. We further show that by transferring knowledge between target architectures with different design constraints, Apollo is able to find optimal configurations faster and often with better objective value (up to 25% improvements). This encouraging outcome portrays a promising path forward to facilitate generating higher quality accelerators.

10 pages, 5 figures, Accepted to Workshop on ML for Systems at the 34th Conference on Neural Information Processing Systems (NeurIPS 2020)

References in corpus (8)

• Practical Bayesian Optimization of Machine Learning Algorithms
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Automatically designing CNN architectures using genetic algorithm for image classification
• Transfer Learning based Dynamic Multiobjective Optimization Algorithms
• A Survey on Practical Applications of Multi-Armed and Contextual Bandits
• Accelerator-aware Neural Network Design using AutoML
• AutoAccel: Automated Accelerator Generation and Optimization with Composable, Parallel and Pipeline Architecture
• Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization

Cited by in corpus (4)

• A Survey of Machine Learning for Computer Architecture and Systems
• A Full-Stack Search Technique for Domain Optimized Deep Learning Accelerators
• AIRCHITECT: Learning Custom Architecture Design and Mapping Space
• Data-Driven Offline Optimization For Architecting Hardware Accelerators