Learning What and Where to Draw

Abstract

Generative Adversarial Networks (GANs) have recently demonstrated the capability to synthesize compelling real-world images, such as room interiors, album covers, manga, faces, birds, and flowers. While existing models can synthesize images based on global constraints such as a class label or caption, they do not provide control over pose or object location. We propose a new model, the Generative Adversarial What-Where Network (GAWWN), that synthesizes images given instructions describing what content to draw in which location. We show high-quality 128 x 128 image synthesis on the Caltech-UCSD Birds dataset, conditioned on both informal text descriptions and also object location. Our system exposes control over both the bounding box around the bird and its constituent parts. By modeling the conditional distributions over part locations, our system also enables conditioning on arbitrary subsets of parts (e.g. only the beak and tail), yielding an efficient interface for picking part locations. We also show preliminary results on the more challenging domain of text- and location-controllable synthesis of images of human actions on the MPII Human Pose dataset.

In NIPS 2016

References in corpus (12)

• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Pixel Recurrent Neural Networks
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• DRAW: A Recurrent Neural Network For Image Generation
• Deep Convolutional Inverse Graphics Network
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• Generative Image Modeling Using Spatial LSTMs
• One-Shot Generalization in Deep Generative Models

Cited by in corpus (81)

• Generative Adversarial Networks: An Overview
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Generative Adversarial Network in Medical Imaging: A Review
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Sharpness-aware Low dose CT denoising using conditional generative adversarial network
• A Review on Deep Learning Techniques for Video Prediction
• Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics
• Adversarial Text-to-Image Synthesis: A Review
• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• Adversarial Networks for Spatial Context-Aware Spectral Image Reconstruction from RGB
• MirrorGAN: Learning Text-to-image Generation by Redescription
• Spatial Evolutionary Generative Adversarial Networks
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Parallel Multiscale Autoregressive Density Estimation
• Object-driven Text-to-Image Synthesis via Adversarial Training
• DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis
• Improving GANs Using Optimal Transport
• Semi-Latent GAN: Learning to generate and modify facial images from attributes
• DiverGAN: An Efficient and Effective Single-Stage Framework for Diverse Text-to-Image Generation
• Generating Multiple Objects at Spatially Distinct Locations
• Multi-Channel Attention Selection GAN with Cascaded Semantic Guidance for Cross-View Image Translation
• Disentangled Person Image Generation
• Using Scene Graph Context to Improve Image Generation
• Unsupervised Person Image Generation with Semantic Parsing Transformation
• Attention-Guided Generative Adversarial Networks for Unsupervised Image-to-Image Translation
• Gesture-to-Gesture Translation in the Wild via Category-Independent Conditional Maps
• LayoutVAE: Stochastic Scene Layout Generation From a Label Set
• Local Class-Specific and Global Image-Level Generative Adversarial Networks for Semantic-Guided Scene Generation
• FTGAN: A Fully-trained Generative Adversarial Networks for Text to Face Generation
• Text2Scene: Generating Compositional Scenes from Textual Descriptions
• Lightweight Generative Adversarial Networks for Text-Guided Image Manipulation
• A Survey and Taxonomy of Adversarial Neural Networks for Text-to-Image Synthesis
• Realistic Image Generation using Region-phrase Attention
• Learning Spatial Pyramid Attentive Pooling in Image Synthesis and Image-to-Image Translation
• Semantic Bottleneck Scene Generation
• Diverse Image Synthesis from Semantic Layouts via Conditional IMLE
• Image Generation from Layout
• PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions
• Label-Noise Robust Generative Adversarial Networks
• cGANs with Conditional Convolution Layer
• Improving Consistency and Correctness of Sequence Inpainting using Semantically Guided Generative Adversarial Network
• Remember What You have drawn: Semantic Image Manipulation with Memory
• Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
• Dual Adversarial Inference for Text-to-Image Synthesis
• Dual Generator Generative Adversarial Networks for Multi-Domain Image-to-Image Translation
• XRayGAN: Consistency-preserving Generation of X-ray Images from Radiology Reports
• Level generation and style enhancement -- deep learning for game development overview
• VITAL: A Visual Interpretation on Text with Adversarial Learning for Image Labeling
• DTGAN: Dual Attention Generative Adversarial Networks for Text-to-Image Generation
• Conditional Generative Adversarial Networks for Emoji Synthesis with Word Embedding Manipulation
• New Ideas and Trends in Deep Multimodal Content Understanding: A Review
• Text Guided Person Image Synthesis
• Deep Attentive Generative Adversarial Network for Photo-Realistic Image De-Quantization
• Attribute-Guided Sketch Generation
• MemeFaceGenerator: Adversarial Synthesis of Chinese Meme-face from Natural Sentences
• Generative Adversarial Networks and Adversarial Autoencoders: Tutorial and Survey
• Multivariate-Information Adversarial Ensemble for Scalable Joint Distribution Matching
• Learning More with Less: Conditional PGGAN-based Data Augmentation for Brain Metastases Detection Using Highly-Rough Annotation on MR Images
• Unselfie: Translating Selfies to Neutral-pose Portraits in the Wild
• Imaginative Walks: Generative Random Walk Deviation Loss for Improved Unseen Learning Representation
• Towards Better Adversarial Synthesis of Human Images from Text
• DLGAN: Disentangling Label-Specific Fine-Grained Features for Image Manipulation
• Inverting and Understanding Object Detectors
• Deep Image Synthesis from Intuitive User Input: A Review and Perspectives
• Words as Art Materials: Generating Paintings with Sequential GANs
• Improving Text to Image Generation using Mode-seeking Function
• Channel Decomposition into Painting Actions
• 3D-MOV: Audio-Visual LSTM Autoencoder for 3D Reconstruction of Multiple Objects from Video
• Visual-Relation Conscious Image Generation from Structured-Text
• Fostering Diversity in Spatial Evolutionary Generative Adversarial Networks
• Trace-Norm Adversarial Examples
• Illiterate DALL-E Learns to Compose
• Inner Space Preserving Generative Pose Machine
• Adversarial Code Learning for Image Generation
• Pathology-Aware Generative Adversarial Networks for Medical Image Augmentation
• Adversarial Synthesis of Human Pose from Text
• Expression Conditional GAN for Facial Expression-to-Expression Translation
• ReshapeGAN: Object Reshaping by Providing A Single Reference Image
• Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text
• CIGLI: Conditional Image Generation from Language & Image
• Learning to Reconstruct and Segment 3D Objects