Direction is what you need: Improving Word Embedding Compression in Large Language Models

Abstract

The adoption of Transformer-based models in natural language processing (NLP) has led to great success using a massive number of parameters. However, due to deployment constraints in edge devices, there has been a rising interest in the compression of these models to improve their inference time and memory footprint. This paper presents a novel loss objective to compress token embeddings in the Transformer-based models by leveraging an AutoEncoder architecture. More specifically, we emphasize the importance of the direction of compressed embeddings with respect to original uncompressed embeddings. The proposed method is task-agnostic and does not require further language modeling pre-training. Our method significantly outperforms the commonly used SVD-based matrix-factorization approach in terms of initial language model Perplexity. Moreover, we evaluate our proposed approach over SQuAD v1.1 dataset and several downstream tasks from the GLUE benchmark, where we also outperform the baseline in most scenarios. Our code is public.

References in corpus (14)

• Efficient Estimation of Word Representations in Vector Space
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Binarized Neural Networks
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Pruning Filters for Efficient ConvNets
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression
• LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression