Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies

Abstract

Learning continuous representations of discrete objects such as text, users, movies, and URLs lies at the heart of many applications including language and user modeling. When using discrete objects as input to neural networks, we often ignore the underlying structures (e.g., natural groupings and similarities) and embed the objects independently into individual vectors. As a result, existing methods do not scale to large vocabulary sizes. In this paper, we design a simple and efficient embedding algorithm that learns a small set of anchor embeddings and a sparse transformation matrix. We call our method Anchor & Transform (ANT) as the embeddings of discrete objects are a sparse linear combination of the anchors, weighted according to the transformation matrix. ANT is scalable, flexible, and end-to-end trainable. We further provide a statistical interpretation of our algorithm as a Bayesian nonparametric prior for embeddings that encourages sparsity and leverages natural groupings among objects. By deriving an approximate inference algorithm based on Small Variance Asymptotics, we obtain a natural extension that automatically learns the optimal number of anchors instead of having to tune it as a hyperparameter. On text classification, language modeling, and movie recommendation benchmarks, we show that ANT is particularly suitable for large vocabulary sizes and demonstrates stronger performance with fewer parameters (up to 40x compression) as compared to existing compression baselines.

arXiv commentICLR 2021, code can be found at http://github.com/pliang279/sparse_discrete

References in corpus (13)

• Distributed Representations of Words and Phrases and their Compositionality
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Exploring the Limits of Language Modeling
• Compressing Neural Networks with the Hashing Trick
• A Fast and Simple Algorithm for Training Neural Probabilistic Language Models
• Nonparametric Bayesian sparse factor models with application to gene expression modeling
• MAD-Bayes: MAP-based Asymptotic Derivations from Bayes
• Compression of Recurrent Neural Networks for Efficient Language Modeling
• Learning K-way D-dimensional Discrete Codes for Compact Embedding Representations
• Hash Embeddings for Efficient Word Representations
• Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
• Compressing Neural Language Models by Sparse Word Representations
• Word Embedding based on Low-Rank Doubly Stochastic Matrix Decomposition