Recurrent Neural Networks: An Embedded Computing Perspective

Abstract

Recurrent Neural Networks (RNNs) are a class of machine learning algorithms used for applications with time-series and sequential data. Recently, there has been a strong interest in executing RNNs on embedded devices. However, difficulties have arisen because RNN requires high computational capability and a large memory space. In this paper, we review existing implementations of RNN models on embedded platforms and discuss the methods adopted to overcome the limitations of embedded systems. We will define the objectives of mapping RNN algorithms on embedded platforms and the challenges facing their realization. Then, we explain the components of RNN models from an implementation perspective. We also discuss the optimizations applied to RNNs to run efficiently on embedded platforms. Finally, we compare the defined objectives with the implementations and highlight some open research questions and aspects currently not addressed for embedded RNNs. Overall, applying algorithmic optimizations to RNN models and decreasing the memory access overhead is vital to obtain high efficiency. To further increase the implementation efficiency, we point up the more promising optimizations that could be applied in future research. Additionally, this article observes that high performance has been targeted by many implementations, while flexibility has, as yet, been attempted less often. Thus, the article provides some guidelines for RNN hardware designers to support flexibility in a better manner.

arXiv commentAccepted for publication in IEEE Access

References in corpus (34)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Unsupervised Learning of Video Representations using LSTMs
• Recent Trends in Deep Learning Based Natural Language Processing
• Binarized Neural Networks
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Ternary Weight Networks
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Tensorizing Neural Networks
• Recent Advances in Recurrent Neural Networks
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Quasi-Recurrent Neural Networks
• SCALE-Sim: Systolic CNN Accelerator Simulator
• Learning a Driving Simulator
• Exploring Sparsity in Recurrent Neural Networks
• EIE: Efficient Inference Engine on Compressed Deep Neural Network
• Block-Sparse Recurrent Neural Networks
• Alternating Multi-bit Quantization for Recurrent Neural Networks
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
• Recurrent Neural Networks With Limited Numerical Precision
• Delta Networks for Optimized Recurrent Network Computation
• DeepIoT: Compressing Deep Neural Network Structures for Sensing Systems with a Compressor-Critic Framework
• Towards Binary-Valued Gates for Robust LSTM Training
• Recurrent Neural Networks With Limited Numerical Precision
• Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networks
• C-LSTM: Enabling Efficient LSTM using Structured Compression Techniques on FPGAs
• RNNFast: An Accelerator for Recurrent Neural Networks Using Domain Wall Memory
• MobiRNN: Efficient Recurrent Neural Network Execution on Mobile GPU
• Single Stream Parallelization of Recurrent Neural Networks for Low Power and Fast Inference

Cited by in corpus (6)

• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
• Review: Deep Learning in Electron Microscopy
• In-flight Novelty Detection with Convolutional Neural Networks
• An Unsupervised Deep Explainable AI Framework for Localization of Concurrent Replay Attacks in Nuclear Reactor Signals
• Deep Learning Approach for Aggressive Driving Behaviour Detection
• Dynamic and Systematic Survey of Deep Learning Approaches for Driving Behavior Analysis