Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations

Abstract

Average word embeddings are a common baseline for more sophisticated sentence embedding techniques. However, they typically fall short of the performances of more complex models such as InferSent. Here, we generalize the concept of average word embeddings to power mean word embeddings. We show that the concatenation of different types of power mean word embeddings considerably closes the gap to state-of-the-art methods monolingually and substantially outperforms these more complex techniques cross-lingually. In addition, our proposed method outperforms different recently proposed baselines such as SIF and Sent2Vec by a solid margin, thus constituting a much harder-to-beat monolingual baseline. Our data and code are publicly available.

arXiv commentExperiments/plots added: Normalization + Figure 1 (dimensionality vs. performance)

References in corpus (1)

• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

Cited by in corpus (28)

• BERTScore: Evaluating Text Generation with BERT
• Survey on reinforcement learning for language processing
• Multilingual Alignment of Contextual Word Representations
• Evaluation of sentence embeddings in downstream and linguistic probing tasks
• A Tutorial on Deep Latent Variable Models of Natural Language
• MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
• Inducing Language-Agnostic Multilingual Representations
• SWE2: SubWord Enriched and Significant Word Emphasized Framework for Hate Speech Detection
• SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models
• Investigating an approach for low resource language dataset creation, curation and classification: Setswana and Sepedi
• Classification from Positive, Unlabeled and Biased Negative Data
• Sentence Analogies: Exploring Linguistic Relationships and Regularities in Sentence Embeddings
• Improving Document Classification with Multi-Sense Embeddings
• Text Embeddings for Retrieval From a Large Knowledge Base
• On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation
• Degree-Aware Alignment for Entities in Tail
• CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model
• Analyzing Structures in the Semantic Vector Space: A Framework for Decomposing Word Embeddings
• EigenEmo: Spectral Utterance Representation Using Dynamic Mode Decomposition for Speech Emotion Classification
• Efficient Sentence Embedding via Semantic Subspace Analysis
• Parameter-free Sentence Embedding via Orthogonal Basis
• Sentence Embeddings for Russian NLU
• Neural Duplicate Question Detection without Labeled Training Data
• Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training
• Vec2Sent: Probing Sentence Embeddings with Natural Language Generation
• A novel sentence embedding based topic detection method for micro-blog
• Discrete Cosine Transform as Universal Sentence Encoder
• Emu: Enhancing Multilingual Sentence Embeddings with Semantic Specialization