Unsupervised Alignment of Embeddings with Wasserstein Procrustes

Abstract

We consider the task of aligning two sets of points in high dimension, which has many applications in natural language processing and computer vision. As an example, it was recently shown that it is possible to infer a bilingual lexicon, without supervised data, by aligning word embeddings trained on monolingual data. These recent advances are based on adversarial training to learn the mapping between the two embeddings. In this paper, we propose to use an alternative formulation, based on the joint estimation of an orthogonal matrix and a permutation matrix. While this problem is not convex, we propose to initialize our optimization algorithm by using a convex relaxation, traditionally considered for the graph isomorphism problem. We propose a stochastic algorithm to minimize our cost function on large scale problems. Finally, we evaluate our method on the problem of unsupervised word translation, by aligning word embeddings trained on monolingual data. On this task, our method obtains state of the art results, while requiring less computational resources than competing approaches.

References in corpus (12)

• Exploiting Similarities among Languages for Machine Translation
• Enriching Word Vectors with Subword Information
• Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration
• Word Translation Without Parallel Data
• Unsupervised Learning by Predicting Noise
• Computational Optimal Transport
• Amplitude and phase variation of point processes
• Wasserstein Discriminant Analysis
• Identifying Word Translations in Non-Parallel Texts
• Non-Adversarial Unsupervised Word Translation
• Graph matching: relax or not?
• Optimal Transport: Fast Probabilistic Approximation with Exact Solvers

Cited by in corpus (15)

• Gromov-Wasserstein Learning for Graph Matching and Node Embedding
• Unsupervised Hyperalignment for Multilingual Word Embeddings
• CONE-Align: Consistent Network Alignment with Proximity-Preserving Node Embedding
• Statistical bounds for entropic optimal transport: sample complexity and the central limit theorem
• Unsupervised Multimodal Representation Learning across Medical Images and Reports
• Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion
• A stable parareal-like method for the second order wave equation
• Unsupervised Multilingual Alignment using Wasserstein Barycenter
• Learning Unsupervised Word Mapping by Maximizing Mean Discrepancy
• Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces
• MosAIc: Finding Artistic Connections across Culture with Conditional Image Retrieval
• Unsupervised Text Summarization via Mixed Model Back-Translation
• Density Matching for Bilingual Word Embedding
• Lost in Evaluation: Misleading Benchmarks for Bilingual Dictionary Induction
• Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings