Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques

Abstract

Deep learning is pervasive in our daily life, including self-driving cars, virtual assistants, social network services, healthcare services, face recognition, etc. However, deep neural networks demand substantial compute resources during training and inference. The machine learning community has mainly focused on model-level optimizations such as architectural compression of deep learning models, while the system community has focused on implementation-level optimization. In between, various arithmetic-level optimization techniques have been proposed in the arithmetic community. This article provides a survey on resource-efficient deep learning techniques in terms of model-, arithmetic-, and implementation-level techniques and identifies the research gaps for resource-efficient deep learning techniques across the three different level techniques. Our survey clarifies the influence from higher to lower-level techniques based on our resource-efficiency metric definition and discusses the future trend for resource-efficient deep learning research.

arXiv commentSubmitted to ACM Computing Surveys

References in corpus (14)

• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices
• A Survey of Neuromorphic Computing and Neural Networks in Hardware
• Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks
• Training and Inference with Integers in Deep Neural Networks
• Fast Training of Convolutional Networks through FFTs
• NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps
• NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision
• Image Classification at Supercomputer Scale
• Dissecting the Graphcore IPU Architecture via Microbenchmarking
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• Performance-Efficiency Trade-off of Low-Precision Numerical Formats in Deep Neural Networks
• Efficient ConvNets for Analog Arrays