Not All Neural Embeddings are Born Equal

Abstract

Neural language models learn word representations that capture rich linguistic and conceptual information. Here we investigate the embeddings learned by neural machine translation models. We show that translation-based embeddings outperform those learned by cutting-edge monolingual models at single-language tasks requiring knowledge of conceptual similarity and/or syntactic role. The findings suggest that, while monolingual models learn information about how concepts are related, neural-translation models better capture their true ontological status.

4 pages plus 1 page of references

References in corpus (4)

• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Natural Language Processing (almost) from Scratch
• From Frequency to Meaning: Vector Space Models of Semantics

Cited by in corpus (9)

• The Geometry of Culture: Analyzing Meaning through Word Embeddings
• A Dual Embedding Space Model for Document Ranking
• Multichannel Variable-Size Convolution for Sentence Classification
• Word and Document Embeddings based on Neural Network Approaches
• Intuitive Contrasting Map for Antonym Embeddings
• Direct Network Transfer: Transfer Learning of Sentence Embeddings for Semantic Similarity
• New word analogy corpus for exploring embeddings of Czech words
• Intrinsic analysis for dual word embedding space models
• Representation Learning for Words and Entities