Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings

Abstract

Recent studies have proposed different methods to improve multilingual word representations in contextualized settings including techniques that align between source and target embedding spaces. For contextualized embeddings, alignment becomes more complex as we additionally take context into consideration. In this work, we propose using Optimal Transport (OT) as an alignment objective during fine-tuning to further improve multilingual contextualized representations for downstream cross-lingual transfer. This approach does not require word-alignment pairs prior to fine-tuning that may lead to sub-optimal matching and instead learns the word alignments within context in an unsupervised manner. It also allows different types of mappings due to soft matching between source and target sentences. We benchmark our proposed method on two tasks (XNLI and XQuAD) and achieve improvements over baselines as well as competitive results compared to similar recent works.

References in corpus (12)

• Exploiting Similarities among Languages for Machine Translation
• On the Cross-lingual Transferability of Monolingual Representations
• Multilingual Alignment of Contextual Word Representations
• Optimal Transport for structured data with application on graphs
• Interpolating between Optimal Transport and MMD using Sinkhorn Divergences
• Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances
• Towards Optimal Transport with Global Invariances
• Unsupervised Alignment of Embeddings with Wasserstein Procrustes
• Screening Sinkhorn Algorithm for Regularized Optimal Transport
• Minibatch optimal transport distances; analysis and applications
• Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing
• Do Explicit Alignments Robustly Improve Multilingual Encoders?