Rethinking Co-design of Neural Architectures and Hardware Accelerators

Abstract

Neural architectures and hardware accelerators have been two driving forces for the progress in deep learning. Previous works typically attempt to optimize hardware given a fixed model architecture or model architecture given fixed hardware. And the dominant hardware architecture explored in this prior work is FPGAs. In our work, we target the optimization of hardware and software configurations on an industry-standard edge accelerator. We systematically study the importance and strategies of co-designing neural architectures and hardware accelerators. We make three observations: 1) the software search space has to be customized to fully leverage the targeted hardware architecture, 2) the search for the model architecture and hardware architecture should be done jointly to achieve the best of both worlds, and 3) different use cases lead to very different search outcomes. Our experiments show that the joint search method consistently outperforms previous platform-aware neural architecture search, manually crafted models, and the state-of-the-art EfficientNet on all latency targets by around 1% on ImageNet top-1 accuracy. Our method can reduce energy consumption of an edge accelerator by up to 2x under the same accuracy constraint, when co-adapting the model architecture and hardware accelerator configurations.

References in corpus (11)

• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• DARTS: Differentiable Architecture Search
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Automatically designing CNN architectures using genetic algorithm for image classification
• Searching for Activation Functions
• Accelerator-aware Neural Network Design using AutoML
• AutoAccel: Automated Accelerator Generation and Optimization with Composable, Parallel and Pipeline Architecture
• MobileDets: Searching for Object Detection Architectures for Mobile Accelerators
• Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Tasks
• Can weight sharing outperform random architecture search? An investigation with TuNAS
• Co-Design of Deep Neural Nets and Neural Net Accelerators for Embedded Vision Applications

Cited by in corpus (5)

• A Full-Stack Search Technique for Domain Optimized Deep Learning Accelerators
• Fast offset corrected in-memory training
• Trends in Energy Estimates for Computing in AI/Machine Learning Accelerators, Supercomputers, and Compute-Intensive Applications
• HALF: Holistic Auto Machine Learning for FPGAs
• NAX: Co-Designing Neural Network and Hardware Architecture for Memristive Xbar based Computing Systems