Compressing Recurrent Neural Network with Tensor Train

Abstract

Recurrent Neural Network (RNN) are a popular choice for modeling temporal and sequential tasks and achieve many state-of-the-art performance on various complex problems. However, most of the state-of-the-art RNNs have millions of parameters and require many computational resources for training and predicting new data. This paper proposes an alternative RNN model to reduce the number of parameters significantly by representing the weight parameters based on Tensor Train (TT) format. In this paper, we implement the TT-format representation for several RNN architectures such as simple RNN and Gated Recurrent Unit (GRU). We compare and evaluate our proposed RNN model with uncompressed RNN model on sequence classification and sequence prediction tasks. Our proposed RNNs with TT-format are able to preserve the performance while reducing the number of RNN parameters significantly up to 40 times smaller.

Accepted at IJCNN 2017

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Do Deep Nets Really Need to be Deep?
• Deep Learning with Limited Numerical Precision
• Predicting Parameters in Deep Learning
• Tensorizing Neural Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Recurrent Neural Network Training with Dark Knowledge Transfer

Cited by in corpus (39)

• Rethinking Bayesian Learning for Data Analysis: The Art of Prior and Inference in Sparsity-Aware Modeling
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Tensorized Embedding Layers for Efficient Model Compression
• Compression of Recurrent Neural Networks for Efficient Language Modeling
• Tensor Train Neighborhood Preserving Embedding
• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• Kronecker CP Decomposition with Fast Multiplication for Compressing RNNs
• Compressing 3DCNNs Based on Tensor Train Decomposition
• Connecting Weighted Automata and Recurrent Neural Networks through Spectral Learning
• On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression
• Run-Time Efficient RNN Compression for Inference on Edge Devices
• Tensor networks for interpretable and efficient quantum-inspired machine learning
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
• Quantum State Tomography for Matrix Product Density Operators
• Restricted Recurrent Neural Networks
• QTN-VQC: An End-to-End Learning framework for Quantum Neural Networks
• Quantization-Aware and Tensor-Compressed Training of Transformers for Natural Language Understanding
• Tensor Decomposition for Compressing Recurrent Neural Network
• Artificial neural networks condensation: A strategy to facilitate adaption of machine learning in medical settings by reducing computational burden
• How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection
• Compressing Language Models using Doped Kronecker Products
• GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking
• Compressing LSTM Networks by Matrix Product Operators
• AdaDeep: A Usage-Driven, Automated Deep Model Compression Framework for Enabling Ubiquitous Intelligent Mobiles
• Convolutional Neural Networks with Transformed Input based on Robust Tensor Network Decomposition
• On the Memory Mechanism of Tensor-Power Recurrent Models
• Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators
• Bayesian Sparsification of Gated Recurrent Neural Networks
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices
• Spectral Pruning for Recurrent Neural Networks
• Tensor train decompositions on recurrent networks
• A Fully Tensorized Recurrent Neural Network
• 3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration
• Block-term Tensor Neural Networks
• Neural network compression via learnable wavelet transforms
• Tensor Methods for Generating Compact Uncertainty Quantification and Deep Learning Models
• Large-Scale Learning with Fourier Features and Tensor Decompositions