UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
arXiv:2111.10023
Abstract
In this paper, we propose a single UniFied transfOrmer (UFO), which is capable of processing either unimodal inputs (e.g., image or language) or multimodal inputs (e.g., the concatenation of the image and the question), for vision-language (VL) representation learning. Existing approaches typically design an individual network for each modality and/or a specific fusion network for multimodal tasks. To simplify the network architecture, we use a single transformer network and enforce multi-task learning during VL pre-training, which includes the image-text contrastive loss, image-text matching loss, and masked language modeling loss based on the bidirectional and the seq2seq attention mask. The same transformer network is used as the image encoder, the text encoder, or the fusion network in different pre-training tasks. Empirically, we observe less conflict among different tasks and achieve new state of the arts on visual question answering, COCO image captioning (cross-entropy optimization) and nocaps (in SPICE). On other downstream tasks, e.g., image-text retrieval, we also achieve competitive performance.
References in corpus (13)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Learning Transferable Visual Models From Natural Language Supervision
- MLP-Mixer: An all-MLP Architecture for Vision
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- Unifying Vision-and-Language Tasks via Text Generation
- Visual Entailment: A Novel Task for Fine-Grained Image Understanding
- A Fast Proximal Point Method for Computing Exact Wasserstein Distance
- MiniVLM: A Smaller and Faster Vision-Language Model
- MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding