Weakly supervised cross-domain alignment with optimal transport
arXiv:2008.06597
Abstract
Cross-domain alignment between image objects and text sequences is key to many visual-language tasks, and it poses a fundamental challenge to both computer vision and natural language processing. This paper investigates a novel approach for the identification and optimization of fine-grained semantic similarities between image and text entities, under a weakly-supervised setup, improving performance over state-of-the-art solutions. Our method builds upon recent advances in optimal transport (OT) to resolve the cross-domain matching problem in a principled manner. Formulated as a drop-in regularizer, the proposed OT solution can be efficiently computed and used in combination with other existing approaches. We present empirical evidence to demonstrate the effectiveness of our approach, showing how it enables simpler model architectures to outperform or be comparable with more sophisticated designs on a range of vision-language tasks.
Accepted to BMVC 2020 (Oral)
References in corpus (12)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Generative Adversarial Text to Image Synthesis
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- VQA: Visual Question Answering
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
- UNITER: UNiversal Image-TExt Representation Learning
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
- Computational Optimal Transport
- A Fast Proximal Point Method for Computing Exact Wasserstein Distance
- Adversarial Text Generation via Feature-Mover's Distance