Generative Visual Manipulation on the Natural Image Manifold

Abstract

Realistic image manipulation is challenging because it requires modifying the image appearance in a user-controlled way, while preserving the realism of the result. Unless the user has considerable artistic skill, it is easy to "fall off" the manifold of natural images while editing. In this paper, we propose to learn the natural image manifold directly from data using a generative adversarial neural network. We then define a class of image editing operations, and constrain their output to lie on that learned manifold at all times. The model automatically adjusts the output keeping all edits as realistic as possible. All our manipulations are expressed in terms of constrained optimization and are applied in near-real time. We evaluate our algorithm on the task of realistic photo manipulation of shape and color. The presented method can further be used for changing one image to look like the other, as well as generating novel imagery from scratch based on user's scribbles.

In European Conference on Computer Vision (ECCV 2016)

References in corpus (2)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

Cited by in corpus (59)

• Generative Adversarial Networks: An Overview
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
• CyCADA: Cycle-Consistent Adversarial Domain Adaptation
• Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
• Neural Photo Editing with Introspective Adversarial Networks
• Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
• Toward Controlled Generation of Text
• Deep Generative Adversarial Networks for Compressed Sensing Automates MRI
• Recent Advances in Deep Learning: An Overview
• Controlling generative models with continuous factors of variations
• Photographic Image Synthesis with Cascaded Refinement Networks
• Video Frame Interpolation via Adaptive Separable Convolution
• FrankenGAN: Guided Detail Synthesis for Building Mass-Models Using Style-Synchonized GANs
• GeneGAN: Learning Object Transfiguration and Attribute Subspace from Unpaired Data
• Activation Maximization Generative Adversarial Nets
• Multi-View Image Generation from a Single-View
• Video Frame Interpolation via Adaptive Convolution
• Context-aware Synthesis for Video Frame Interpolation
• Real-Time User-Guided Image Colorization with Learned Deep Priors
• AlphaGAN: Generative adversarial networks for natural image matting
• Scribbler: Controlling Deep Image Synthesis with Sketch and Color
• VITON: An Image-based Virtual Try-on Network
• JointGAN: Multi-Domain Joint Distribution Learning with Generative Adversarial Nets
• Inverting The Generator Of A Generative Adversarial Network (II)
• SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing
• Latent Convolutional Models
• Conditional Image-to-Image Translation
• Message Passing Multi-Agent GANs
• A Provably Convergent Scheme for Compressive Sensing under Random Generative Priors
• Every Smile is Unique: Landmark-Guided Diverse Smile Generation
• Ranking CGANs: Subjective Control over Semantic Image Attributes
• Solving Linear Inverse Problems Using GAN Priors: An Algorithm with Provable Guarantees
• AlignGAN: Learning to Align Cross-Domain Images with Conditional Generative Adversarial Networks
• Disentangling Multiple Conditional Inputs in GANs
• Interactive Image Manipulation with Natural Language Instruction Commands
• X-GANs: Image Reconstruction Made Easy for Extreme Cases
• Generative Melody Composition with Human-in-the-Loop Bayesian Optimization
• Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructions
• PixelNN: Example-based Image Synthesis
• Unpaired Photo-to-Caricature Translation on Faces in the Wild
• Model Extraction and Defenses on Generative Adversarial Networks
• Unpaired Multi-Domain Image Generation via Regularized Conditional GANs
• Towards Recovery of Conditional Vectors from Conditional Generative Adversarial Networks
• r-BTN: Cross-domain Face Composite and Synthesis from Limited Facial Patches
• Towards an Understanding of Neural Networks in Natural-Image Spaces
• Bayesian Reasoning with Trained Neural Networks
• Color Sails: Discrete-Continuous Palettes for Deep Color Exploration
• Generalized Latent Variable Recovery for Generative Adversarial Networks
• GANtruth - an unpaired image-to-image translation method for driving scenarios
• Generative Creativity: Adversarial Learning for Bionic Design
• Excavate Condition-invariant Space by Intrinsic Encoder
• Inverse mapping of face GANs
• Out-of-Sample Testing for GANs
• Guiding the One-to-one Mapping in CycleGAN via Optimal Transport
• Reconstructing Images of Two Adjacent Objects through Scattering Medium Using Generative Adversarial Network
• KGAN: How to Break The Minimax Game in GAN
• BSD-GAN: Branched Generative Adversarial Network for Scale-Disentangled Representation Learning and Image Synthesis
• Instance Map based Image Synthesis with a Denoising Generative Adversarial Network