Scaling Up Deep Neural Network Optimization for Edge Inference

Abstract

Deep neural networks (DNNs) have been increasingly deployed on and integrated with edge devices, such as mobile phones, drones, robots and wearables. To run DNN inference directly on edge devices (a.k.a. edge inference) with a satisfactory performance, optimizing the DNN design (e.g., network architecture and quantization policy) is crucial. While state-of-the-art DNN designs have leveraged performance predictors to speed up the optimization process, they are device-specific (i.e., each predictor for only one target device) and hence cannot scale well in the presence of extremely diverse edge devices. Moreover, even with performance predictors, the optimizer (e.g., search-based optimization) can still be time-consuming when optimizing DNNs for many different devices. In this work, we propose two approaches to scaling up DNN optimization. In the first approach, we reuse the performance predictors built on a proxy device, and leverage the performance monotonicity to scale up the DNN optimization without re-building performance predictors for each different device. In the second approach, we build scalable performance predictors that can estimate the resulting performance (e.g., inference accuracy/latency/energy) given a DNN-device pair, and use a neural network-based automated optimizer that takes both device features and optimization parameters as input and then directly outputs the optimal DNN design without going through a lengthy optimization process for each individual device.

Position paper. New algorithm added. Part of the content (from Section 5 "Learning to Optimize") will be presented in the work-in-progress poster session of the ACM/IEEE Symposium on Edge Computing 2020

References in corpus (11)

• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Neural Architecture Optimization
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• Embedded Binarized Neural Networks
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• On Neural Architecture Search for Resource-Constrained Hardware Platforms
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Evaluating Efficient Performance Estimators of Neural Architectures
• APQ: Joint Search for Network Architecture, Pruning and Quantization Policy
• Generative Design of Hardware-aware DNNs