Swivel: Improving Embeddings by Noticing What's Missing

Abstract

We present Submatrix-wise Vector Embedding Learner (Swivel), a method for generating low-dimensional feature embeddings from a feature co-occurrence matrix. Swivel performs approximate factorization of the point-wise mutual information matrix via stochastic gradient descent. It uses a piecewise loss with special handling for unobserved co-occurrences, and thus makes use of all the information in the matrix. While this requires computation proportional to the size of the entire matrix, we make use of vectorized multiplication to process thousands of rows and columns at once to compute millions of predicted values. Furthermore, we partition the matrix into shards in order to parallelize the computation across many nodes. This approach results in more accurate embeddings than can be achieved with methods that consider only observed co-occurrences, and can scale to much larger corpora than can be handled with sampling methods.

arXiv comment9 pages, 4 figures

References in corpus (3)

• Efficient Estimation of Word Representations in Vector Space
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation

Cited by in corpus (17)

• PyTorch-BigGraph: A Large-scale Graph Embedding System
• Efficient Natural Language Response Suggestion for Smart Reply
• Characterization of citizens using word2vec and latent topic analysis in a large set of tweets
• Learning Crosslingual Word Embeddings without Bilingual Corpora
• Bridging the Gap for Tokenizer-Free Language Models
• Vector representations of text data in deep learning
• Deconstructing and reconstructing word embedding algorithms
• Relaxed Softmax for learning from Positive and Unlabeled data
• Specializing Joint Representations for the task of Product Recommendation
• On the Impact of Knowledge-based Linguistic Annotations in the Quality of Scientific Embeddings
• Word Embedding Algorithms as Generalized Low Rank Models and their Canonical Form
• Why So Down? The Role of Negative (and Positive) Pointwise Mutual Information in Distributional Semantics
• Lessons Learned from Applying off-the-shelf BERT: There is no Silver Bullet
• Adversarial Training of Word2Vec for Basket Completion
• Item-Item Music Recommendations With Side Information
• Single Training Dimension Selection for Word Embedding with PCA
• Evaluating Word Embedding Hyper-Parameters for Similarity and Analogy Tasks