Efficient Vector Representation for Documents through Corruption

Abstract

We present an efficient document representation learning framework, Document Vector through Corruption (Doc2VecC). Doc2VecC represents each document as a simple average of word embeddings. It ensures a representation generated as such captures the semantic meanings of the document during learning. A corruption model is included, which introduces a data-dependent regularization that favors informative or rare words while forcing the embeddings of common and non-discriminative ones to be close to zero. Doc2VecC produces significantly better word embeddings than Word2Vec. We compare Doc2VecC with several state-of-the-art document representation learning algorithms. The simple model architecture introduced by Doc2VecC matches or out-performs the state-of-the-art in generating high-quality document representations for sentiment analysis, document classification as well as semantic relatedness tasks. The simplicity of the model enables training on billions of words per hour on a single machine. At the same time, the model is very efficient in generating representations of unseen documents at test time.

5th International Conference on Learning Representations, 2017

References in corpus (5)

• Text Understanding from Scratch
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Document Embedding with Paragraph Vectors
• Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews
• Multi-Step Regression Learning for Compositional Distributional Semantics

Cited by in corpus (17)

• Neural Vector Spaces for Unsupervised Information Retrieval
• Improving Hospital Mortality Prediction with Medical Named Entities and Multimodal Learning
• Extracting Sentence Embeddings from Pretrained Transformer Models
• On the Importance of Attention in Meta-Learning for Few-Shot Text Classification
• KHAN: Knowledge-Aware Hierarchical Attention Networks for Accurate Political Stance Prediction
• Learning Unsupervised Semantic Document Representation for Fine-grained Aspect-based Sentiment Analysis
• Improving Visually Grounded Sentence Representations with Self-Attention
• Multimodal Deep Learning for Mental Disorders Prediction from Audio Speech Samples
• Unsupervised Document Embedding via Contrastive Augmentation
• Learning Representations from Product Titles for Modeling Shopping Transactions
• Iterative Relevance Feedback for Answer Passage Retrieval with Passage-level Semantic Match
• Parameter-Efficient Neural Question Answering Models via Graph-Enriched Document Representations
• Measuring a Texts Fairness Dimensions Using Machine Learning Based on Social Psychological Factors
• Effective Distributed Representations for Academic Expert Search
• Paraphrase Thought: Sentence Embedding Module Imitating Human Language Recognition
• SNU_IDS at SemEval-2018 Task 12: Sentence Encoder with Contextualized Vectors for Argument Reasoning Comprehension
• Improving Clinical Outcome Predictions Using Convolution over Medical Entities with Multimodal Learning