Swapping Autoencoder for Deep Image Manipulation

Abstract

Deep generative models have become increasingly effective at producing realistic images from randomly sampled seeds, but using such models for controllable manipulation of existing images remains challenging. We propose the Swapping Autoencoder, a deep model designed specifically for image manipulation, rather than random sampling. The key idea is to encode an image with two independent components and enforce that any swapped combination maps to a realistic image. In particular, we encourage the components to represent structure and texture, by enforcing one component to encode co-occurrent patch statistics across different parts of an image. As our method is trained with an encoder, finding the latent codes for a new input image becomes trivial, rather than cumbersome. As a result, it can be used to manipulate real input images in various ways, including texture swapping, local and global editing, and latent code vector arithmetic. Experiments on multiple datasets show that our model produces better results and is substantially more efficient compared to recent generative models.

arXiv commentNeurIPS 2020. Please visit https://taesung.me/SwappingAutoencoder/ for an introductory video. v2 mainly contains reorganization of the Introduction and Broader Impact section

References in corpus (18)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Instance Normalization: The Missing Ingredient for Fast Stylization
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Which Training Methods for GANs do actually Converge?
• Disentangling by Factorising
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Invertible Conditional GANs for image editing
• Making Convolutional Networks Shift-Invariant Again
• Disentangling factors of variation in deep representations using adversarial training
• Non-Stationary Texture Synthesis by Adversarial Expansion
• Learning visual groups from co-occurrences in space and time
• Learning Hierarchical Semantic Image Manipulation through Structured Representations
• High-Resolution Daytime Translation Without Domain Labels
• Multi-mapping Image-to-Image Translation via Learning Disentanglement
• Disentangling Content and Style via Unsupervised Geometry Distillation

Cited by in corpus (11)

• PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
• GANSlider: How Users Control Generative Models for Images using Multiple Sliders with and without Feedforward Information
• Learning to Generate 3D Shapes from a Single Example
• Reinforcing Generated Images via Meta-learning for One-Shot Fine-Grained Visual Recognition
• Image-to-Image Translation: Methods and Applications
• Neural Knitworks: Patched Neural Implicit Representation Networks
• Dual In-painting Model for Unsupervised Gaze Correction and Animation in the Wild
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Learning Compositional Representation for 4D Captures with Neural ODE
• VidStyleODE: Disentangled Video Editing via StyleGAN and NeuralODEs
• Variational Bayesian Framework for Advanced Image Generation with Domain-Related Variables