X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers

Abstract

Mirroring the success of masked language models, vision-and-language counterparts like ViLBERT, LXMERT and UNITER have achieved state of the art performance on a variety of multimodal discriminative tasks like visual question answering and visual grounding. Recent work has also successfully adapted such models towards the generative task of image captioning. This begs the question: Can these models go the other way and generate images from pieces of text? Our analysis of a popular representative from this model family - LXMERT - finds that it is unable to generate rich and semantically meaningful imagery with its current training setup. We introduce X-LXMERT, an extension to LXMERT with training refinements including: discretizing visual representations, using uniform masking with a large range of masking ratios and aligning the right pre-training datasets to the right objectives which enables it to paint. X-LXMERT's image generation capabilities rival state of the art generative models while its question answering and captioning abilities remains comparable to LXMERT. Finally, we demonstrate the generality of these training refinements by adding image generation capabilities into UNITER to produce X-UNITER.

EMNLP 2020

References in corpus (13)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Decoupled Weight Decay Regularization
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Gaussian Error Linear Units (GELUs)
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Neural Discrete Representation Learning
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Pixel Recurrent Neural Networks
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• Controllable Text-to-Image Generation
• Selfie: Self-supervised Pretraining for Image Embedding
• A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models

Cited by in corpus (10)

• Zero-Shot Text-to-Image Generation
• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
• Unifying Multimodal Transformer for Bi-directional Image and Text Generation
• Rethinking the constraints of multimodal fusion: case study in Weakly-Supervised Audio-Visual Video Parsing
• Multi-Tailed, Multi-Headed, Spatial Dynamic Memory refined Text-to-Image Synthesis
• Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text
• Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization
• Exploration into Translation-Equivariant Image Quantization
• CIGLI: Conditional Image Generation from Language & Image