Grow and Prune Compact, Fast, and Accurate LSTMs

Abstract

Long short-term memory (LSTM) has been widely used for sequential data modeling. Researchers have increased LSTM depth by stacking LSTM cells to improve performance. This incurs model redundancy, increases run-time delay, and makes the LSTMs more prone to overfitting. To address these problems, we propose a hidden-layer LSTM (H-LSTM) that adds hidden layers to LSTM's original one level non-linear control gates. H-LSTM increases accuracy while employing fewer external stacked layers, thus reducing the number of parameters and run-time latency significantly. We employ grow-and-prune (GP) training to iteratively adjust the hidden layers through gradient-based growth and magnitude-based pruning of connections. This learns both the weights and the compact architecture of H-LSTM control gates. We have GP-trained H-LSTMs for image captioning and speech recognition applications. For the NeuralTalk architecture on the MSCOCO dataset, our three models reduce the number of parameters by 38.7x [floating-point operations (FLOPs) by 45.5x], run-time latency by 4.5x, and improve the CIDEr score by 2.6. For the DeepSpeech2 architecture on the AN4 dataset, our two models reduce the number of parameters by 19.4x (FLOPs by 23.5x), run-time latency by 15.7%, and the word error rate from 12.9% to 8.7%. Thus, GP-trained H-LSTMs can be seen to be compact, fast, and accurate.

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Recurrent Neural Network Regularization
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Compressing Deep Convolutional Networks using Vector Quantization
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Compressing Neural Networks with the Hashing Trick
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• How to Construct Deep Recurrent Neural Networks
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• DSD: Dense-Sparse-Dense Training for Deep Neural Networks
• Exploring Sparsity in Recurrent Neural Networks
• Learning Compact Recurrent Neural Networks
• Image Captioning

Cited by in corpus (11)

• Sparse Networks from Scratch: Faster Training without Losing Performance
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation
• One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation
• Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM
• Incremental Learning Using a Grow-and-Prune Paradigm with Efficient Neural Networks
• Artificial neural networks condensation: A strategy to facilitate adaption of machine learning in medical settings by reducing computational burden
• Topological Insights into Sparse Neural Networks
• Image Captioning with Sparse Recurrent Neural Network
• DiabDeep: Pervasive Diabetes Diagnosis based on Wearable Medical Sensors and Efficient Neural Networks