FLASH: Fast Neural Architecture Search with Hardware Optimization

Abstract

Neural architecture search (NAS) is a promising technique to design efficient and high-performance deep neural networks (DNNs). As the performance requirements of ML applications grow continuously, the hardware accelerators start playing a central role in DNN design. This trend makes NAS even more complicated and time-consuming for most real applications. This paper proposes FLASH, a very fast NAS methodology that co-optimizes the DNN accuracy and performance on a real hardware platform. As the main theoretical contribution, we first propose the NN-Degree, an analytical metric to quantify the topological characteristics of DNNs with skip connections (e.g., DenseNets, ResNets, Wide-ResNets, and MobileNets). The newly proposed NN-Degree allows us to do training-free NAS within one second and build an accuracy predictor by training as few as 25 samples out of a vast search space with more than 63 billion configurations. Second, by performing inference on the target hardware, we fine-tune and validate our analytical models to estimate the latency, area, and energy consumption of various DNN architectures while executing standard ML datasets. Third, we construct a hierarchical algorithm based on simplicial homology global optimization (SHGO) to optimize the model-architecture co-design process, while considering the area, latency, and energy consumption of the target hardware. We demonstrate that, compared to the state-of-the-art NAS approaches, our proposed hierarchical SHGO-based algorithm enables more than four orders of magnitude speedup (specifically, the execution time of the proposed algorithm is about 0.1 seconds). Finally, our experimental evaluations show that FLASH is easily transferable to different hardware architectures, thus enabling us to do NAS on a Raspberry Pi-3B processor in less than 3 seconds.

arXiv commentPublished at ACM CODES+ISSS 2021

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Playing Atari with Deep Reinforcement Learning
• Neural Architecture Search with Reinforcement Learning
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Language Models are Few-Shot Learners
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DARTS: Differentiable Architecture Search
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• Large-Scale Evolution of Image Classifiers
• MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
• BRP-NAS: Prediction-based NAS using GCNs
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS