Controllable and Progressive Image Extrapolation

Abstract

Image extrapolation aims at expanding the narrow field of view of a given image patch. Existing models mainly deal with natural scene images of homogeneous regions and have no control of the content generation process. In this work, we study conditional image extrapolation to synthesize new images guided by the input structured text. The text is represented as a graph to specify the objects and their spatial relation to the unknown regions of the image. Inspired by drawing techniques, we propose a progressive generative model of three stages, i.e., generating a coarse bounding-boxes layout, refining it to a finer segmentation layout, and mapping the layout to a realistic output. Such a multi-stage design is shown to facilitate the training process and generate more controllable results. We validate the effectiveness of the proposed method on the face and human clothing dataset in terms of visual results, quantitative evaluations and flexible controls.

References in corpus (8)

• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
• Non-Stationary Texture Synthesis by Adversarial Expansion
• Visual Dynamics: Stochastic Future Generation via Layered Cross Convolutional Networks
• Foreground-aware Image Inpainting
• Learning to Multi-Task by Active Sampling

Cited by in corpus (6)

• Text as Neural Operator: Image Manipulation by Text Instruction
• Understanding the Role of Scene Graphs in Visual Question Answering
• Neural Design Network: Graphic Layout Generation with Constraints
• RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval
• ReGO: Reference-Guided Outpainting for Scenery Image
• GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks