Controllable Image Synthesis via SegVAE
arXiv:2007.08397
Abstract
Flexible user controls are desirable for content creation and image editing. A semantic map is commonly used intermediate representation for conditional image generation. Compared to the operation on raw RGB pixels, the semantic map enables simpler user modification. In this work, we specifically target at generating semantic maps given a label-set consisting of desired categories. The proposed framework, SegVAE, synthesizes semantic maps in an iterative manner using conditional variational autoencoder. Quantitative and qualitative experiments demonstrate that the proposed model can generate realistic and diverse semantic maps. We also apply an off-the-shelf image-to-image translation model to generate realistic RGB images to better understand the quality of the synthesized semantic maps. Furthermore, we showcase several real-world image-editing applications including object removal, object insertion, and object replacement.
arXiv commentECCV 2020. Project page: https://yccyenchicheng.github.io/SegVAE/ Code: https://github.com/yccyenchicheng/SegVAE
References in corpus (11)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Auto-Encoding Variational Bayes
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Conditional Image Generation with PixelCNN Decoders
- Pixel Recurrent Neural Networks
- Deep Human Parsing with Active Template Regression
- Stochastic Video Generation with a Learned Prior
- MaskGAN: Towards Diverse and Interactive Facial Image Manipulation
- Context-Aware Synthesis and Placement of Object Instances
- Spatially Controllable Image Synthesis with Internal Representation Collaging
- Automatic Objects Removal for Scene Completion