Offline bilingual word vectors, orthogonal transformations and the inverted softmax

Abstract

Usually bilingual word vectors are trained "online". Mikolov et al. showed they can also be found "offline", whereby two pre-trained embeddings are aligned with a linear transformation, using dictionaries compiled from expert knowledge. In this work, we prove that the linear transformation between two spaces should be orthogonal. This transformation can be obtained using the singular value decomposition. We introduce a novel "inverted softmax" for identifying translation pairs, with which we improve the precision @1 of Mikolov's original mapping from 34% to 43%, when translating a test set composed of both common and rare English words into Italian. Orthogonal transformations are more robust to noise, enabling us to learn the transformation without expert bilingual signal by constructing a "pseudo-dictionary" from the identical character strings which appear in both languages, achieving 40% precision on the same test set. Finally, we extend our method to retrieve the true translations of English sentences from a corpus of 200k Italian sentences with a precision @1 of 68%.

arXiv commentAccepted to conference track at ICLR 2017

Cited by in corpus (56)

• Multilingual Alignment of Contextual Word Representations
• A Call for More Rigor in Unsupervised Cross-lingual Learning
• Predicting Concreteness and Imageability of Words Within and Across Languages via Word Embeddings
• Time-aware Graph Neural Networks for Entity Alignment between Temporal Knowledge Graphs
• Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information
• From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers
• Unsupervised Cross-lingual Adaptation for Sequence Tagging and Beyond
• Emerging Cross-lingual Structure in Pretrained Language Models
• A Novel Bilingual Word Embedding Method for Lexical Translation Using Bilingual Sense Clique
• On the Robustness of Unsupervised and Semi-supervised Cross-lingual Word Embedding Learning
• Revisiting the Context Window for Cross-lingual Word Embeddings
• Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing
• Robust Cross-lingual Embeddings from Parallel Sentences
• Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing
• Unsupervised Multilingual Alignment using Wasserstein Barycenter
• A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
• Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework
• Bilingual Dictionary Based Neural Machine Translation without Using Parallel Sentences
• Cross-Lingual Word Embedding Refinement by ${\ell }_1$ Norm Optimisation
• Filtered Inner Product Projection for Crosslingual Embedding Alignment
• What makes multilingual BERT multilingual?
• Zero-Shot Cross-Lingual Opinion Target Extraction
• Learning Multilingual Word Embeddings Using Image-Text Data
• On Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing
• Two-Level Transformer and Auxiliary Coherence Modeling for Improved Text Segmentation
• Are Girls Neko or Shōjo? Cross-Lingual Alignment of Non-Isomorphic Embeddings with Iterative Normalization
• Cross-Lingual Syntactic Transfer through Unsupervised Adaptation of Invertible Projections
• Multi-source Learning via Completion of Block-wise Overlapping Noisy Matrices
• Do Explicit Alignments Robustly Improve Multilingual Encoders?
• Learning Cross-lingual Embeddings from Twitter via Distant Supervision
• Aligning Vector-spaces with Noisy Supervised Lexicons
• HAL: Improved Text-Image Matching by Mitigating Visual Semantic Hubs
• Cross-lingual Data Transformation and Combination for Text Classification
• Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction
• On a Novel Application of Wasserstein-Procrustes for Unsupervised Cross-Lingual Learning
• Bilingual Lexicon Induction via Unsupervised Bitext Construction and Word Alignment
• Sentence transition matrix: An efficient approach that preserves sentence semantics
• Why Overfitting Isn't Always Bad: Retrofitting Cross-Lingual Word Embeddings to Dictionaries
• Expanding the Text Classification Toolbox with Cross-Lingual Embeddings
• Weakly-Supervised Concept-based Adversarial Learning for Cross-lingual Word Embeddings
• Fast Search with Poor OCR
• A Strong and Robust Baseline for Text-Image Matching
• Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training
• Cross-language Information Retrieval
• Bi-Decoder Augmented Network for Neural Machine Translation
• Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes
• Target Language-Aware Constrained Inference for Cross-lingual Dependency Parsing
• Are We Consistently Biased? Multidimensional Analysis of Biases in Distributional Word Vectors
• Learning Efficient Task-Specific Meta-Embeddings with Word Prisms
• Unsupervised Word Translation Pairing using Refinement based Point Set Registration
• RPD: A Distance Function Between Word Embeddings
• Word Embeddings: Stability and Semantic Change
• Document Network Embedding: Coping for Missing Content and Missing Links
• Revisiting Adversarial Autoencoder for Unsupervised Word Translation with Cycle Consistency and Improved Training
• Improving Unsupervised Word-by-Word Translation with Language Model and Denoising Autoencoder
• Locality Preserving Loss: Neighbors that Live together, Align together