Multilingual Models for Compositional Distributed Semantics

Abstract

We present a novel technique for learning semantic representations, which extends the distributional hypothesis to multilingual data and joint-space embeddings. Our models leverage parallel data and learn to strongly align the embeddings of semantically equivalent sentences, while maintaining sufficient distance between those of dissimilar sentences. The models do not rely on word alignments or any syntactic information and are successfully applied to a number of diverse languages. We extend our approach to learn semantic representations at the document level, too. We evaluate these models on two cross-lingual document classification tasks, outperforming the prior state of the art. Through qualitative analysis and the study of pivoting effects we demonstrate that our representations are semantically plausible and can capture semantic relationships across languages without parallel data.

Proceedings of ACL 2014 (Long papers)

References in corpus (4)

• Exploiting Similarities among Languages for Machine Translation
• Polyglot: Distributed Word Representations for Multilingual NLP
• Recurrent Convolutional Neural Networks for Discourse Compositionality
• Learning Multilingual Word Representations using a Bag-of-Words Autoencoder

Cited by in corpus (35)

• Cross-lingual Language Model Pretraining
• XNLI: Evaluating Cross-lingual Sentence Representations
• Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation
• Separated by an Un-common Language: Towards Judgment Language Informed Vector Space Modeling
• XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering
• Multilingual Hierarchical Attention Networks for Document Classification
• Cross-lingual Models of Word Embeddings: An Empirical Comparison
• Learning Bilingual Word Representations by Marginalizing Alignments
• Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints
• Transfer Learning for Speech and Language Processing
• Learning Crosslingual Word Embeddings without Bilingual Corpora
• Learning to Represent Words in Context with Multilingual Supervision
• Leveraging Monolingual Data for Crosslingual Compositional Word Representations
• Crosslingual Document Embedding as Reduced-Rank Ridge Regression
• Building Program Vector Representations for Deep Learning
• Improving Cross-Lingual Word Embeddings by Meeting in the Middle
• Part-of-Speech Relevance Weights for Learning Word Embeddings
• Neural Cross-Lingual Named Entity Recognition with Minimal Resources
• Survey on the Use of Typological Information in Natural Language Processing
• Bilingual Learning of Multi-sense Embeddings with Discrete Autoencoders
• Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework
• Unsupervised Sentiment Analysis for Code-mixed Data
• Leveraging Pretrained Word Embeddings for Part-of-Speech Tagging of Code Switching Data
• Filtering and Mining Parallel Data in a Joint Multilingual Space
• Context Aware Machine Learning
• Cross-lingual Dataless Classification for Languages with Small Wikipedia Presence
• Acoustic Feature Learning via Deep Variational Canonical Correlation Analysis
• Correlational Neural Networks
• Language classification from bilingual word embedding graphs
• FUSE: Multi-Faceted Set Expansion by Coherent Clustering of Skip-grams
• Learning Multilingual Topics from Incomparable Corpus
• Bilingual Language Modeling, A transfer learning technique for Roman Urdu
• Multi-lingual Common Semantic Space Construction via Cluster-consistent Word Embedding
• CLUSE: Cross-Lingual Unsupervised Sense Embeddings
• Learning Multilingual Embeddings for Cross-Lingual Information Retrieval in the Presence of Topically Aligned Corpora