On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression
arXiv:1908.09982
Abstract
Despite their ubiquity in NLP tasks, Long Short-Term Memory (LSTM) networks suffer from computational inefficiencies caused by inherent unparallelizable recurrences, which further aggravates as LSTMs require more parameters for larger memory capacity. In this paper, we propose to apply low-rank matrix factorization (MF) algorithms to different recurrences in LSTMs, and explore the effectiveness on different NLP tasks and model components. We discover that additive recurrence is more important than multiplicative recurrence, and explain this by identifying meaningful correlations between matrix norms and compression performance. We compare our approach across two settings: 1) compressing core LSTM recurrences in language models, 2) compressing biLSTM layers of ELMo evaluated in three downstream NLP tasks.
Accepted in PACLIC 2019
References in corpus (13)
- Distilling the Knowledge in a Neural Network
- Sequence to Sequence Learning with Neural Networks
- Bidirectional Attention Flow for Machine Comprehension
- Enhanced LSTM for Natural Language Inference
- Compressing Deep Convolutional Networks using Vector Quantization
- A large annotated corpus for learning natural language inference
- Supervised and Unsupervised Speech Enhancement Using Nonnegative Matrix Factorization
- Learning Intrinsic Sparse Structures within Long Short-Term Memory
- Compressing Recurrent Neural Network with Tensor Train
- Factorization tricks for LSTM networks
- Tensor-Train Recurrent Neural Networks for Video Classification
- Neural Networks Compression for Language Modeling
- Learning to Prune Filters in Convolutional Neural Networks
Cited by in corpus (6)
- Few-Shot Bot: Prompt-Based Learning for Dialogue Systems
- LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression
- Direction is what you need: Improving Word Embedding Compression in Large Language Models
- Greenformer: Factorization Toolkit for Efficient Deep Neural Networks
- A Survey on Green Deep Learning
- Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition