Word Mover's Embedding: From Word2Vec to Document Embedding

Abstract

While the celebrated Word2Vec technique yields semantically rich representations for individual words, there has been relatively less success in extending to generate unsupervised sentences or documents embeddings. Recent work has demonstrated that a distance measure between documents called \emph{Word Mover's Distance} (WMD) that aligns semantically similar words, yields unprecedented KNN classification accuracy. However, WMD is expensive to compute, and it is hard to extend its use beyond a KNN classifier. In this paper, we propose the \emph{Word Mover's Embedding } (WME), a novel approach to building an unsupervised document (sentence) embedding from pre-trained word embeddings. In our experiments on 9 benchmark text classification datasets and 22 textual similarity tasks, the proposed technique consistently matches or outperforms state-of-the-art techniques, with significantly higher accuracy on problems of short length.

arXiv commentEMNLP'18 Camera-Ready Version

References in corpus (11)

• Efficient Estimation of Word Representations in Vector Space
• Exploiting Similarities among Languages for Machine Translation
• Convolutional Neural Networks for Sentence Classification
• Character-Aware Neural Language Models
• Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration
• A Hierarchical Neural Autoencoder for Paragraphs and Documents
• Towards Universal Paraphrastic Sentence Embeddings
• Sentence-State LSTM for Text Representation
• A Recurrent Encoder-Decoder Network for Sequential Face Alignment
• D2KE: From Distance to Kernel and Embedding
• Prepositions in Context

Cited by in corpus (10)

• Evaluation of Text Generation: A Survey
• Cultural Cartography with Word Embeddings
• Exploiting Rich Syntactic Information for Semantic Parsing with Graph-to-Sequence Model
• Speeding up Word Mover's Distance and its variants via properties of distances between embeddings
• Scalable Nearest Neighbor Search for Optimal Transport
• Reinforcement Learning Based Text Style Transfer without Parallel Training Corpus
• An Efficient Shared-memory Parallel Sinkhorn-Knopp Algorithm to Compute the Word Mover's Distance
• Technical Question Answering across Tasks and Domains
• Constructing a Highlight Classifier with an Attention Based LSTM Neural Network
• Corruption Is Not All Bad: Incorporating Discourse Structure into Pre-training via Corruption for Essay Scoring