Cross-Modal Contrastive Learning for Text-to-Image Generation

Abstract

The output of text-to-image synthesis systems should be coherent, clear, photo-realistic scenes with high semantic fidelity to their conditioned text descriptions. Our Cross-Modal Contrastive Generative Adversarial Network (XMC-GAN) addresses this challenge by maximizing the mutual information between image and text. It does this via multiple contrastive losses which capture inter-modality and intra-modality correspondences. XMC-GAN uses an attentional self-modulation generator, which enforces strong text-image correspondence, and a contrastive discriminator, which acts as a critic as well as a feature encoder for contrastive learning. The quality of XMC-GAN's output is a major step up from previous models, as we show on three challenging datasets. On MS-COCO, not only does XMC-GAN improve state-of-the-art FID from 24.70 to 9.33, but--more importantly--people prefer XMC-GAN by 77.3 for image quality and 74.1 for image-text alignment, compared to three other recent models. XMC-GAN also generalizes to the challenging Localized Narratives dataset (which has longer, more detailed descriptions), improving state-of-the-art FID from 48.70 to 14.12. Lastly, we train and evaluate XMC-GAN on the challenging Open Images data, establishing a strong benchmark FID score of 26.91.

arXiv commentCVPR 2021

References in corpus (18)

• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• Generative Adversarial Text to Image Synthesis
• Improved Baselines with Momentum Contrastive Learning
• Conditional Image Generation with PixelCNN Decoders
• What Makes for Good Views for Contrastive Learning?
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• A Note on the Inception Score
• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• Image Augmentations for GAN Training
• ContraGAN: Contrastive Learning for Conditional Image Generation
• Controllable Text-to-Image Generation
• On Self Modulation for Generative Adversarial Networks
• Geometry-Contrastive GAN for Facial Expression Transfer
• Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO

Cited by in corpus (5)

• Adversarial Text-to-Image Synthesis: A Review
• Improving Text-to-Image Synthesis Using Contrastive Learning
• Deep Image Synthesis from Intuitive User Input: A Review and Perspectives
• Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization
• CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization