Optimizing Performance of Recurrent Neural Networks on GPUs

Abstract

As recurrent neural networks become larger and deeper, training times for single networks are rising into weeks or even months. As such there is a significant incentive to improve the performance and scalability of these networks. While GPUs have become the hardware of choice for training and deploying recurrent models, the implementations employed often make use of only basic optimizations for these architectures. In this article we demonstrate that by exposing parallelism between operations within the network, an order of magnitude speedup across a range of network sizes can be achieved over a naive implementation. We describe three stages of optimization that have been incorporated into the fifth release of NVIDIA's cuDNN: firstly optimizing a single cell, secondly a single layer, and thirdly the entire network.

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Exploring the Limits of Language Modeling
• Transition-Based Dependency Parsing with Stack Long Short-Term Memory
• Fast Algorithms for Convolutional Neural Networks
• rnn : Recurrent Library for Torch

Cited by in corpus (22)

• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Privacy-preserving Federated Learning for Residential Short Term Load Forecasting
• Edge Intelligence: Architectures, Challenges, and Applications
• LightRNN: Memory and Computation-Efficient Recurrent Neural Networks
• All SMILES Variational Autoencoder
• TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models
• Wider and Deeper, Cheaper and Faster: Tensorized LSTMs for Sequence Learning
• LSTM Benchmarks for Deep Learning Frameworks
• Fast and Accurate Modeling of Transient-State Gradient-Spoiled Sequences by Recurrent Neural Networks
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Scheduling Computation Graphs of Deep Learning Models on Manycore CPUs
• FusionStitching: Boosting Memory Intensive Computations for Deep Learning Workloads
• UnICORNN: A recurrent model for learning very long time dependencies
• Rich Character-Level Information for Korean Morphological Analysis and Part-of-Speech Tagging
• Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training
• Automatic Horizontal Fusion for GPU Kernels
• Demystifying BERT: Implications for Accelerator Design
• When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
• A Lightweight Recurrent Network for Sequence Modeling
• Sparse Persistent RNNs: Squeezing Large Recurrent Networks On-Chip
• BPPSA: Scaling Back-propagation by Parallel Scan Algorithm
• Learning Sparse Structured Ensembles with SG-MCMC and Network Pruning