Compressing Word Embeddings via Deep Compositional Code Learning

Abstract

Natural language processing (NLP) models often require a massive number of parameters for word embeddings, resulting in a large storage or memory footprint. Deploying neural NLP models to mobile devices requires compressing the word embeddings without any significant sacrifices in performance. For this purpose, we propose to construct the embeddings with few basis vectors. For each word, the composition of basis vectors is determined by a hash code. To maximize the compression rate, we adopt the multi-codebook quantization approach instead of binary coding scheme. Each code is composed of multiple discrete numbers, such as (3, 2, 1, 8), where the value of each component is limited to a fixed range. We propose to directly learn the discrete codes in an end-to-end neural network by applying the Gumbel-softmax trick. Experiments show the compression rate achieves 98% in a sentiment analysis task and 94% ~ 99% in machine translation tasks without performance loss. In both tasks, the proposed method can improve the model performance by slightly lowering the compression rate. Compared to other approaches such as character-level segmentation, the proposed method is language-independent and does not require modifications to the network architecture.

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Character-Aware Neural Language Models
• Compressing Neural Networks with the Hashing Trick
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Learning Structured Sparsity in Deep Neural Networks

Cited by in corpus (16)

• Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
• Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
• Word2Bits - Quantized Word Vectors
• Near-lossless Binarization of Word Embeddings
• DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling
• Learning K-way D-dimensional Discrete Code For Compact Embedding Representations
• Learning Compressed Sentence Representations for On-Device Text Processing
• Convolutional neural network compression for natural language processing
• Deconstructing and reconstructing word embedding algorithms
• General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference
• METEOR: Learning Memory and Time Efficient Representations from Multi-modal Data Streams
• Hamming Sentence Embeddings for Information Retrieval
• Error-Correcting Neural Sequence Prediction
• Unsupervised Domain-adaptive Hash for Networks
• Compositional Coding for Collaborative Filtering
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression