Unifying Multimodal Transformer for Bi-directional Image and Text Generation

Abstract

We study the joint learning of image-to-text and text-to-image generations, which are naturally bi-directional tasks. Typical existing works design two separate task-specific models for each task, which impose expensive design efforts. In this work, we propose a unified image-and-text generative framework based on a single multimodal model to jointly study the bi-directional tasks. We adopt Transformer as our unified architecture for its strong performance and task-agnostic design. Specifically, we formulate both tasks as sequence generation tasks, where we represent images and text as unified sequences of tokens, and the Transformer learns multimodal interactions to generate sequences. We further propose two-level granularity feature representations and sequence-level training to improve the Transformer-based unified framework. Experiments show that our approach significantly improves previous Transformer-based model X-LXMERT's FID from 37.0 to 29.9 (lower is better) for text-to-image generation, and improves CIDEr-D score from 100.9% to 122.6% for fine-tuned image-to-text generation on the MS-COCO dataset. Our code is available online.

arXiv commentACM MM 2021 (Industrial Track). Code: https://github.com/researchmm/generate-it

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Auto-Encoding Variational Bayes
• Learning Transferable Visual Models From Natural Language Supervision
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Categorical Reparameterization with Gumbel-Softmax
• Generative Adversarial Text to Image Synthesis
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Improved Techniques for Training GANs
• Zero-Shot Text-to-Image Generation
• VQA: Visual Question Answering
• Sequence Level Training with Recurrent Neural Networks
• CogView: Mastering Text-to-Image Generation via Transformers
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• A Note on the Inception Score
• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• Aligning Linguistic Words and Visual Semantic Units for Image Captioning
• MirrorGAN: Learning Text-to-image Generation by Redescription
• Attention on Attention for Image Captioning
• DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis
• Turbo Learning for Captionbot and Drawingbot
• X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
• In Defense of Grid Features for Visual Question Answering

Cited by in corpus (3)

• Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment
• Learning Bidirectional Translation between Descriptions and Actions with Small Paired Data
• Exploration into Translation-Equivariant Image Quantization