BilBOWA: Fast Bilingual Distributed Representations without Word Alignments

Abstract

We introduce BilBOWA (Bilingual Bag-of-Words without Alignments), a simple and computationally-efficient model for learning bilingual distributed representations of words which can scale to large monolingual datasets and does not require word-aligned parallel training data. Instead it trains directly on monolingual data and extracts a bilingual signal from a smaller set of raw-text sentence-aligned data. This is achieved using a novel sampled bag-of-words cross-lingual objective, which is used to regularize two noise-contrastive language models for efficient cross-lingual feature learning. We show that bilingual embeddings learned using the proposed model outperform state-of-the-art methods on a cross-lingual document classification task as well as a lexical translation task on WMT11 data.

References in corpus (4)

• Natural Language Processing (almost) from Scratch
• Frustratingly Easy Domain Adaptation
• word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method
• An Autoencoder Approach to Learning Bilingual Word Representations

Cited by in corpus (72)

• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• A Survey Of Cross-lingual Word Embedding Models
• Word Translation Without Parallel Data
• Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• The Missing Ingredient in Zero-Shot Neural Machine Translation
• Unsupervised Neural Machine Translation
• Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation
• Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classification
• XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering
• Analyzing the Limitations of Cross-lingual Word Embedding Mappings
• A Call for More Rigor in Unsupervised Cross-lingual Learning
• Multilingual Hierarchical Attention Networks for Document Classification
• Cross-lingual Models of Word Embeddings: An Empirical Comparison
• Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints
• Vocabulary Selection Strategies for Neural Machine Translation
• Extracting Parallel Sentences with Bidirectional Recurrent Neural Networks to Improve Machine Translation
• Nationality Classification Using Name Embeddings
• A Novel Bilingual Word Embedding Method for Lexical Translation Using Bilingual Sense Clique
• Learning Translations via Matrix Completion
• Learning Crosslingual Word Embeddings without Bilingual Corpora
• Bilingual Embeddings with Random Walks over Multilingual Wordnets
• Efficiently Reusing Natural Language Processing Models for Phenotype-Mention Identification in Free-text Electronic Medical Records: Methodology Study
• Cross-Lingual Sentiment Analysis Without (Good) Translation
• Crosslingual Document Embedding as Reduced-Rank Ridge Regression
• Revisiting the Context Window for Cross-lingual Word Embeddings
• Neural Cross-Lingual Named Entity Recognition with Minimal Resources
• Robust Cross-lingual Embeddings from Parallel Sentences
• Code-switched inspired losses for generic spoken dialog representations
• Unsupervised Multilingual Alignment using Wasserstein Barycenter
• Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework
• Unsupervised Multilingual Word Embeddings
• Document Embedding for Scientific Articles: Efficacy of Word Embeddings vs TFIDF
• A Cross-Architecture Instruction Embedding Model for Natural Language Processing-Inspired Binary Code Analysis
• Chinese Discourse Segmentation Using Bilingual Discourse Commonality
• Subword Segmentation and a Single Bridge Language Affect Zero-Shot Neural Machine Translation
• Cross-Lingual BERT Contextual Embedding Space Mapping with Isotropic and Isometric Conditions
• Multilingual Knowledge Graph Embeddings for Cross-lingual Knowledge Alignment
• Transformer based Multilingual document Embedding model
• Semantic Entity Enrichment by Leveraging Multilingual Descriptions for Link Prediction
• Language classification from bilingual word embedding graphs
• Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction
• Cross-lingual Entity Alignment with Incidental Supervision
• Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings
• Funnelling: A New Ensemble Method for Heterogeneous Transfer Learning and its Application to Cross-Lingual Text Classification
• Neural Task Representations as Weak Supervision for Model Agnostic Cross-Lingual Transfer
• Discovering Bilingual Lexicons in Polyglot Word Embeddings
• Embedding Learning Through Multilingual Concept Induction
• Dimension Projection among Languages based on Pseudo-relevant Documents for Query Translation
• Multilingual Word Embeddings using Multigraphs
• Isomorphic Cross-lingual Embeddings for Low-Resource Languages
• Multilingual Embeddings Jointly Induced from Contexts and Concepts: Simple, Strong and Scalable
• Density Matching for Bilingual Word Embedding
• Learning Bilingual Word Embeddings Using Lexical Definitions
• Traceability Support for Multi-Lingual Software Projects
• A Strong Baseline for Learning Cross-Lingual Word Embeddings from Sentence Alignments
• Beyond Offline Mapping: Learning Cross Lingual Word Embeddings through Context Anchoring
• PESTS: Persian_English Cross Lingual Corpus for Semantic Textual Similarity
• Neural Article Pair Modeling for Wikipedia Sub-article Matching
• Characterizing Departures from Linearity in Word Translation
• Code-switching Language Modeling With Bilingual Word Embeddings: A Case Study for Egyptian Arabic-English
• Multi-lingual Common Semantic Space Construction via Cluster-consistent Word Embedding
• Multi-Task Learning for Argumentation Mining
• Resolving Out-of-Vocabulary Words with Bilingual Embeddings in Machine Translation
• CLUSE: Cross-Lingual Unsupervised Sense Embeddings
• Data Augmentation with Unsupervised Machine Translation Improves the Structural Similarity of Cross-lingual Word Embeddings
• Bilingual Topic Models for Comparable Corpora
• Bilingual Dictionary Induction for Bantu Languages
• NMT-based Cross-lingual Document Embeddings
• Unsupervised Word Translation Pairing using Refinement based Point Set Registration
• Variational learning across domains with triplet information
• Multilevel Text Alignment with Cross-Document Attention
• Learning to Represent Bilingual Dictionaries