Resource-Efficient Neural Networks for Embedded Systems

Abstract

While machine learning is traditionally a resource intensive task, embedded systems, autonomous navigation, and the vision of the Internet of Things fuel the interest in resource-efficient approaches. These approaches aim for a carefully chosen trade-off between performance and resource consumption in terms of computation and energy. The development of such approaches is among the major challenges in current machine learning research and key to ensure a smooth transition of machine learning technology from a scientific environment with virtually unlimited computing resources into everyday's applications. In this article, we provide an overview of the current state of the art of machine learning techniques facilitating these real-world requirements. In particular, we focus on resource-efficient inference based on deep neural networks (DNNs), the predominant machine learning models of the past decade. We give a comprehensive overview of the vast literature that can be mainly split into three non-mutually exclusive categories: (i) quantized neural networks, (ii) network pruning, and (iii) structural efficiency. These techniques can be applied during training or as post-processing, and they are widely used to reduce the computational demands in terms of memory footprint, inference speed, and energy efficiency. We also briefly discuss different concepts of embedded hardware for DNNs and their compatibility with machine learning techniques as well as potential for energy and latency reduction. We substantiate our discussion with experiments on well-known benchmark data sets using compression techniques (quantization, pruning) for a set of resource-constrained embedded systems, such as CPUs, GPUs and FPGAs. The obtained results highlight the difficulty of finding good trade-offs between resource efficiency and prediction quality.

arXiv commentarXiv admin note: text overlap with arXiv:1812.02240; accepted at JMLR

References in corpus (29)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Neural Architecture Search with Reinforcement Learning
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Expectation Propagation for approximate Bayesian inference
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Rethinking the Value of Network Pruning
• Binarized Neural Networks
• Ternary Weight Networks
• Trained Ternary Quantization
• Tensorizing Neural Networks
• Fixed Point Quantization of Deep Convolutional Networks
• Compressing Neural Networks with the Hashing Trick
• Variational Dropout and the Local Reparameterization Trick
• Convolutional Neural Networks using Logarithmic Data Representation
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Training and Inference with Integers in Deep Neural Networks
• Training deep neural networks with low precision multiplications
• Bayesian Dark Knowledge
• Soft Weight-Sharing for Neural Network Compression
• Diversity Networks: Neural Network Compression Using Determinantal Point Processes
• Probabilistic Binary Neural Networks
• ARM: Augment-REINFORCE-Merge Gradient for Stochastic Binary Networks
• $L_0$-ARM: Network Sparsification via Stochastic Binary Optimization
• The High-Dimensional Geometry of Binary Neural Networks

Cited by in corpus (6)

• Resource-efficient Deep Neural Networks for Automotive Radar Interference Mitigation
• A Dimensionality Reduction Approach for Convolutional Neural Networks
• Artificial Neural Networks for Sensor Data Classification on Small Embedded Systems
• Quantized Neural Networks for Radar Interference Mitigation
• Resource-Efficient Speech Mask Estimation for Multi-Channel Speech Enhancement
• Binarized Weight Error Networks With a Transition Regularization Term