Photographic Image Synthesis with Cascaded Refinement Networks

Abstract

We present an approach to synthesizing photographic images conditioned on semantic layouts. Given a semantic label map, our approach produces an image with photographic appearance that conforms to the input layout. The approach thus functions as a rendering engine that takes a two-dimensional semantic specification of the scene and produces a corresponding photographic image. Unlike recent and contemporaneous work, our approach does not rely on adversarial training. We show that photographic images can be synthesized from semantic layouts by a single feedforward network with appropriate structure, trained end-to-end with a direct regression objective. The presented approach scales seamlessly to high resolutions; we demonstrate this by synthesizing photographic images at 2-megapixel resolution, the full resolution of our training data. Extensive perceptual experiments on datasets of outdoor and indoor scenes demonstrate that images synthesized by the presented approach are considerably more realistic than alternative approaches. The results are shown in the supplementary video at https://youtu.be/0fhUJT21-bs

arXiv commentPublished at the International Conference on Computer Vision (ICCV 2017)

References in corpus (19)

• Explaining and Harnessing Adversarial Examples
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Multi-Scale Context Aggregation by Dilated Convolutions
• Improved Techniques for Training GANs
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• DRAW: A Recurrent Neural Network For Image Generation
• Generating Videos with Scene Dynamics
• Fully Convolutional Networks for Semantic Segmentation
• Layer Normalization
• Unrolled Generative Adversarial Networks
• Unsupervised Learning for Physical Interaction through Video Prediction
• Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Generative Visual Manipulation on the Natural Image Manifold
• Attribute2Image: Conditional Image Generation from Visual Attributes
• DeepStereo: Learning to Predict New Views from the World's Imagery
• Generating Images from Captions with Attention
• Learning to Generate Chairs, Tables and Cars with Convolutional Networks

Cited by in corpus (14)

• Single-Image SVBRDF Capture with a Rendering-Aware Deep Network
• 3D Sketching using Multi-View Deep Volumetric Prediction
• Unsupervised Portrait Shadow Removal via Generative Priors
• Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Human-Like Sketches
• Personalized and Occupational-aware Age Progression by Generative Adversarial Networks
• Disentangling Factors of Variation by Mixing Them
• Two-phase Hair Image Synthesis by Self-Enhancing Generative Model
• InverseNet: Solving Inverse Problems with Splitting Networks
• PixelNN: Example-based Image Synthesis
• Pipeline Generative Adversarial Networks for Facial Images Generation with Multiple Attributes
• Learning Generative Models of Tissue Organization with Supervised GANs
• Line Artist: A Multiple Style Sketch to Painting Synthesis Scheme
• Instance Map based Image Synthesis with a Denoising Generative Adversarial Network
• Synthesizing Photorealistic Images with Deep Generative Learning