Image Generation from Layout
arXiv:1811.11389
Abstract
Despite significant recent progress on generative models, controlled generation of images depicting multiple and complex object layouts is still a difficult problem. Among the core challenges are the diversity of appearance a given object may possess and, as a result, exponential set of images consistent with a specified layout. To address these challenges, we propose a novel approach for layout-based image generation; we call it Layout2Im. Given the coarse spatial layout (bounding boxes + object categories), our model can generate a set of realistic images which have the correct objects in the desired locations. The representation of each object is disentangled into a specified/certain part (category) and an unspecified/uncertain part (appearance). The category is encoded using a word embedding and the appearance is distilled into a low-dimensional vector sampled from a normal distribution. Individual object representations are composed together using convolutional LSTM, to obtain an encoding of the complete layout, and then decoded to an image. Several loss terms are introduced to encourage accurate and diverse generation. The proposed Layout2Im model significantly outperforms the previous state of the art, boosting the best reported inception score by 24.66% and 28.57% on the very challenging COCO-Stuff and Visual Genome datasets, respectively. Extensive experiments also demonstrate our method's ability to generate complex and diverse images with multiple objects.
arXiv commentAccepted to CVPR 2019 (Oral)
References in corpus (24)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Auto-Encoding Variational Bayes
- Conditional Generative Adversarial Nets
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Spectral Normalization for Generative Adversarial Networks
- Conditional Image Generation with PixelCNN Decoders
- Pixel Recurrent Neural Networks
- Toward Multimodal Image-to-Image Translation
- High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
- Modulating early visual processing by language
- Disentangling factors of variation in deep representations using adversarial training
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- Learning What and Where to Draw
- DRIT++: Diverse Image-to-Image Translation via Disentangled Representations
- Discovering Hidden Factors of Variation in Deep Networks
- Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
- ChatPainter: Improving Text to Image Generation using Dialogue
- Generating Images from Captions with Attention
- Parallel Multiscale Autoregressive Density Estimation
- COCO-Stuff: Thing and Stuff Classes in Context
- CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
- Text2Scene: Generating Compositional Scenes from Textual Descriptions