Tensorized Embedding Layers for Efficient Model Compression

Abstract

The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing. However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting. We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.

References in corpus (26)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Sequence to Sequence Learning with Neural Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Tensorizing Neural Networks
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Pointer Sentinel Mixture Models
• Variational Dropout and the Local Reparameterization Trick
• Matrix product operator representations
• Tensor Ring Decomposition
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Compressing Recurrent Neural Network with Tensor Train
• Ultimate tensorization: compressing convolutional and FC layers alike
• A Tensorized Transformer for Language Modeling
• Long-term Forecasting using Higher Order Tensor RNNs
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• Tensor-Train Recurrent Neural Networks for Video Classification
• Adaptive Input Representations for Neural Language Modeling
• Learning K-way D-dimensional Discrete Codes for Compact Embedding Representations
• Wide Compression: Tensor Ring Nets
• Word2Bits - Quantized Word Vectors
• Bayesian Sparsification of Recurrent Neural Networks
• Learning K-way D-dimensional Discrete Code For Compact Embedding Representations
• GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking

Cited by in corpus (19)

• Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
• Self-Supervised Learning for Data Scarcity in a Fatigue Damage Prognostic Problem
• Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
• Tensor Decomposition for Model Reduction in Neural Networks: A Review
• A Model Compression Method with Matrix Product Operators for Speech Enhancement
• Mixed-Precision Embedding Using a Cache
• TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models
• KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation
• On-FPGA Training with Ultra Memory Reduction: A Low-Precision Tensor Method
• Training with Multi-Layer Embeddings for Model Reduction
• Tensorized Transformer for Dynamical Systems Modeling
• Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems
• On the Downstream Performance of Compressed Word Embeddings
• Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators
• 3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration
• A Fully Tensorized Recurrent Neural Network
• Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition
• MARS: Masked Automatic Ranks Selection in Tensor Decompositions
• Weight Squeezing: Reparameterization for Knowledge Transfer and Model Compression