All-in-One Image-Grounded Conversational Agents
arXiv:1912.12394
Abstract
As single-task accuracy on individual language and image tasks has improved substantially in the last few years, the long-term goal of a generally skilled agent that can both see and talk becomes more feasible to explore. In this work, we focus on leveraging individual language and image tasks, along with resources that incorporate both vision and language towards that objective. We design an architecture that combines state-of-the-art Transformer and ResNeXt modules fed into a novel attentive multimodal module to produce a combined model trained on many tasks. We provide a thorough analysis of the components of the model, and transfer performance when training on one, some, or all of the tasks. Our final models provide a single system that obtains good results on all vision and language tasks considered, and improves the state-of-the-art in image-grounded conversational applications.
References in corpus (12)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Microsoft COCO Captions: Data Collection and Evaluation Server
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- The Natural Language Decathlon: Multitask Learning as Question Answering
- UNITER: UNiversal Image-TExt Representation Learning
- Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
- Pythia v0.1: the Winning Entry to the VQA Challenge 2018
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
- Stacked Cross Attention for Image-Text Matching
- Real or Fake? Learning to Discriminate Machine from Human Generated Text
- Training Millions of Personalized Dialogue Agents