Representation learning for very short texts using weighted word embedding aggregation

Abstract

Short text messages such as tweets are very noisy and sparse in their use of vocabulary. Traditional textual representations, such as tf-idf, have difficulty grasping the semantic meaning of such texts, which is important in applications such as event detection, opinion mining, news recommendation, etc. We constructed a method based on semantic word embeddings and frequency information to arrive at low-dimensional representations for short texts designed to capture semantic similarity. For this purpose we designed a weight-based model and a learning procedure based on a novel median-based loss function. This paper discusses the details of our model and the optimization methods, together with the experimental results on both Wikipedia and Twitter data. We find that our method outperforms the baseline approaches in the experiments, and that it generalizes well on different word embeddings without retraining. Our method is therefore capable of retaining most of the semantic information in the text, and is applicable out-of-the-box.

8 pages, 3 figures, 2 tables, appears in Pattern Recognition Letters

References in corpus (9)

• Efficient Estimation of Word Representations in Vector Space
• Sequence to Sequence Learning with Neural Networks
• Natural Language Processing (almost) from Scratch
• Distributed Representations of Sentences and Documents
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• A Convolutional Neural Network for Modelling Sentences
• Semi-supervised Convolutional Neural Networks for Text Categorization via Region Embedding
• Learning Semantic Similarity for Very Short Texts
• N-gram-Based Low-Dimensional Representation for Document Classification

Cited by in corpus (21)

• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• Pretrained Transformers for Text Ranking: BERT and Beyond
• The Use of NLP-Based Text Representation Techniques to Support Requirement Engineering Tasks: A Systematic Mapping Review
• Sentiment analysis based on rhetorical structure theory: Learning deep neural networks from discourse trees
• A novel methodology on distributed representations of proteins using their interacting ligands
• Tag Embedding Based Personalized Point Of Interest Recommendation System
• Large-Scale User Modeling with Recurrent Neural Networks for Music Discovery on Multiple Time Scales
• An Effective Automatic Image Annotation Model Via Attention Model and Data Equilibrium
• Large-scale Collaborative Filtering with Product Embeddings
• Probabilistic Label Trees for Extreme Multi-label Classification
• Correlation Coefficients and Semantic Textual Similarity
• Merchandise Recommendation for Retail Events with Word Embedding Weighted Tf-idf and Dynamic Query Expansion
• Real-time Visual Object Tracking with Natural Language Description
• Attention improves concentration when learning node embeddings
• Word-Emoji Embeddings from large scale Messaging Data reflect real-world Semantic Associations of Expressive Icons
• CEIL: A General Classification-Enhanced Iterative Learning Framework for Text Clustering
• Cross-Discourse and Multilingual Exploration of Textual Corpora with the DualNeighbors Algorithm
• Learning Sentence Embeddings for Coherence Modelling and Beyond
• Semantic Representation and Inference for NLP
• Correlations between Word Vector Sets
• Hierarchical Bayesian Personalized Recommendation: A Case Study and Beyond