Deep Image Synthesis from Intuitive User Input: A Review and Perspectives

Abstract

In many applications of computer graphics, art and design, it is desirable for a user to provide intuitive non-image input, such as text, sketch, stroke, graph or layout, and have a computer system automatically generate photo-realistic images that adhere to the input content. While classic works that allow such automatic image content generation have followed a framework of image retrieval and composition, recent advances in deep generative models such as generative adversarial networks (GANs), variational autoencoders (VAEs), and flow-based methods have enabled more powerful and versatile image generation tasks. This paper reviews recent works for image synthesis given intuitive user input, covering advances in input versatility, image generation methodology, benchmark datasets, and evaluation metrics. This motivates new perspectives on input representation and interactivity, cross pollination between major image generation paradigms, and evaluation and comparison of generation methods.

arXiv comment26 pages, 7 figures, 1 table

References in corpus (25)

• Conditional Generative Adversarial Nets
• Semi-Supervised Classification with Graph Convolutional Networks
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Self-Attention Generative Adversarial Networks
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Variational Inference with Normalizing Flows
• Density estimation using Real NVP
• Deep Human Parsing with Active Template Regression
• Learning What and Where to Draw
• Semantic Understanding of Scenes through the ADE20K Dataset
• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
• Semantic Jitter: Dense Supervision for Visual Comparisons via Synthetic Images
• Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis
• Generating Images from Captions with Attention
• Learning Latent Subspaces in Variational Autoencoders
• Convolutional Sketch Inversion
• Cross-Modal Contrastive Learning for Text-to-Image Generation
• Generating Multiple Objects at Spatially Distinct Locations
• Text2Scene: Generating Compositional Scenes from Textual Descriptions