Learning Multi-granular Quantized Embeddings for Large-Vocab Categorical Features in Recommender Systems
arXiv:2002.08530
Abstract
Recommender system models often represent various sparse features like users, items, and categorical features via embeddings. A standard approach is to map each unique feature value to an embedding vector. The size of the produced embedding table grows linearly with the size of the vocabulary. Therefore, a large vocabulary inevitably leads to a gigantic embedding table, creating two severe problems: (i) making model serving intractable in resource-constrained environments; (ii) causing overfitting problems. In this paper, we seek to learn highly compact embeddings for large-vocab sparse features in recommender systems (recsys). First, we show that the novel Differentiable Product Quantization (DPQ) approach can generalize to recsys problems. In addition, to better handle the power-law data distribution commonly seen in recsys, we propose a Multi-Granular Quantized Embeddings (MGQE) technique which learns more compact embeddings for infrequent items. We seek to provide a new angle to improve recommendation performance with compact model sizes. Extensive experiments on three recommendation tasks and two datasets show that we can achieve on par or better performance, with only ~20% of the original model size.
arXiv commentlonger version
References in corpus (10)
- Distributed Representations of Words and Phrases and their Compositionality
- Distilling the Knowledge in a Neural Network
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems
- Compressing Neural Networks with the Hashing Trick
- Translation-based Recommendation
- Learning K-way D-dimensional Discrete Codes for Compact Embedding Representations
- Hash Embeddings for Efficient Word Representations
- Differentiable Product Quantization for End-to-End Embedding Compression
- Candidate Generation with Binary Codes for Large-Scale Top-N Recommendation