Adversarial Text-to-Image Synthesis: A Review

Abstract

With the advent of generative adversarial networks, synthesizing images from textual descriptions has recently become an active research area. It is a flexible and intuitive way for conditional image generation with significant progress in the last years regarding visual realism, diversity, and semantic alignment. However, the field still faces several challenges that require further research efforts such as enabling the generation of high-resolution images with multiple objects, and developing suitable and reliable evaluation metrics that correlate with human judgement. In this review, we contextualize the state of the art of adversarial text-to-image synthesis models, their development since their inception five years ago, and propose a taxonomy based on the level of supervision. We critically examine current strategies to evaluate text-to-image synthesis models, highlight shortcomings, and identify new areas of research, ranging from the development of better datasets and evaluation metrics to possible improvements in architectural design and model training. This review complements previous surveys on generative adversarial networks with a focus on text-to-image synthesis which we believe will help researchers to further advance the field.

Published at Neural Networks Journal, available at https://www.sciencedirect.com/science/article/pii/S0893608021002823

References in corpus (74)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Neural Machine Translation by Jointly Learning to Align and Translate
• Conditional Generative Adversarial Nets
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Generative Adversarial Networks: An Overview
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification
• Conditional Image Generation with PixelCNN Decoders
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Pixel Recurrent Neural Networks
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• A Structured Self-attentive Sentence Embedding
• NICE: Non-linear Independent Components Estimation
• Score-Based Generative Modeling through Stochastic Differential Equations
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Zero-Shot Text-to-Image Generation
• Generative Modeling by Estimating Gradients of the Data Distribution
• DRAW: A Recurrent Neural Network For Image Generation
• A Learned Representation For Artistic Style
• Skip-Thought Vectors
• End-To-End Memory Networks
• Adversarially Learned Inference
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• A note on the evaluation of generative models
• Large Scale Adversarial Representation Learning
• How Generative Adversarial Networks and Their Variants Work: An Overview
• Improved Precision and Recall Metric for Assessing Generative Models
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• Image Transformer
• A Note on the Inception Score
• Learning What and Where to Draw
• Semantic Photo Manipulation with a Generative Image Prior
• Assessing Generative Models via Precision and Recall
• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• Classification Accuracy Score for Conditional Generative Models
• Learning Deep Representations of Fine-grained Visual Descriptions
• TAC-GAN - Text Conditioned Auxiliary Classifier Generative Adversarial Network
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models
• ChatPainter: Improving Text to Image Generation using Dialogue
• Controllable Text-to-Image Generation
• Generating Images from Captions with Attention
• Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language
• PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph
• Reliable Fidelity and Diversity Metrics for Generative Models
• Text-Guided Neural Image Inpainting
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Parallel Multiscale Autoregressive Density Estimation
• Cross-Modal Contrastive Learning for Text-to-Image Generation
• On the Evaluation of Conditional GANs
• Generating Multiple Objects at Spatially Distinct Locations
• Adversarial score matching and improved sampling for image generation
• Interactive Image Generation Using Scene Graphs
• Controlling Style and Semantics in Weakly-Supervised Image Generation
• Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis
• Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO
• From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speech
• Network-to-Network Translation with Conditional Invertible Neural Networks
• Realistic Image Generation using Region-phrase Attention
• Semantically Invariant Text-to-Image Generation
• Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approach
• Leveraging Visual Question Answering to Improve Text-to-Image Synthesis
• Faces à la Carte: Text-to-Face Generation via Attribute Disentanglement
• Conditional Image Generation and Manipulation for User-Specified Content
• C4Synth: Cross-Caption Cycle-Consistent Text-to-Image Synthesis
• Text-to-Image Generation Grounded by Fine-Grained User Attention
• S2IGAN: Speech-to-Image Generation via Adversarial Learning

Cited by in corpus (9)

• A survey of the Vision Transformers and their CNN-Transformer based Variants
• Diffusion Models, Image Super-Resolution And Everything: A Survey
• CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders
• Hitchhiker's Guide to Super-Resolution: Introduction and Recent Advances
• Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations
• PromptMap: An Alternative Interaction Style for AI-Based Image Generation
• ClimateGAN: Raising Climate Change Awareness by Generating Images of Floods
• Offline Evaluation of Set-Based Text-to-Image Generation
• TransAnaNet: Transformer-based Anatomy Change Prediction Network for Head and Neck Cancer Patient Radiotherapy