Learning K-way D-dimensional Discrete Codes for Compact Embedding Representations

Abstract

Conventional embedding methods directly associate each symbol with a continuous embedding vector, which is equivalent to applying a linear transformation based on a "one-hot" encoding of the discrete symbols. Despite its simplicity, such approach yields the number of parameters that grows linearly with the vocabulary size and can lead to overfitting. In this work, we propose a much more compact K-way D-dimensional discrete encoding scheme to replace the "one-hot" encoding. In the proposed "KD encoding", each symbol is represented by a $D$-dimensional code with a cardinality of $K$, and the final symbol embedding vector is generated by composing the code embedding vectors. To end-to-end learn semantically meaningful codes, we derive a relaxed discrete optimization approach based on stochastic gradient descent, which can be generally applied to any differentiable computational graph with an embedding layer. In our experiments with various applications from natural language processing to graph convolutional networks, the total size of the embedding layer can be reduced up to 98\% while achieving similar or better performance.

arXiv commentICML 2018. arXiv admin note: text overlap with arXiv:1711.03067

Cited by in corpus (17)

• Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
• Tensorized Embedding Layers for Efficient Model Compression
• Near-lossless Binarization of Word Embeddings
• Learning Compressed Sentence Representations for On-Device Text Processing
• Superbloom: Bloom filter meets Transformer
• Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies
• Saec: Similarity-Aware Embedding Compression in Recommendation Systems
• Deep Retrieval: Learning A Retrievable Structure for Large-Scale Recommendations
• NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashing
• Doubly Sparse: Sparse Mixture of Sparse Experts for Efficient Softmax Inference
• On the Downstream Performance of Compressed Word Embeddings
• Learning Multi-granular Quantized Embeddings for Large-Vocab Categorical Features in Recommender Systems
• Discrete Word Embedding for Logical Natural Language Understanding
• Unsupervised Domain-adaptive Hash for Networks
• All Word Embeddings from One Embedding
• Latent Multi-Criteria Ratings for Recommendations
• Depthwise Discrete Representation Learning