COCO-GAN: Generation by Parts via Conditional Coordinating

Abstract

Humans can only interact with part of the surrounding environment due to biological restrictions. Therefore, we learn to reason the spatial relationships across a series of observations to piece together the surrounding environment. Inspired by such behavior and the fact that machines also have computational constraints, we propose \underline{CO}nditional \underline{CO}ordinate GAN (COCO-GAN) of which the generator generates images by parts based on their spatial coordinates as the condition. On the other hand, the discriminator learns to justify realism across multiple assembled patches by global coherence, local appearance, and edge-crossing continuity. Despite the full images are never generated during training, we show that COCO-GAN can produce \textbf{state-of-the-art-quality} full images during inference. We further demonstrate a variety of novel applications enabled by teaching the network to be aware of coordinates. First, we perform extrapolation to the learned coordinate manifold and generate off-the-boundary patches. Combining with the originally generated full image, COCO-GAN can produce images that are larger than training samples, which we called "beyond-boundary generation". We then showcase panorama generation within a cylindrical coordinate system that inherently preserves horizontally cyclic topology. On the computation side, COCO-GAN has a built-in divide-and-conquer paradigm that reduces memory requisition during training and inference, provides high-parallelism, and can generate parts of images on-demand.

Accepted to ICCV'19 (oral). All images are compressed due to size limit, please access the full-resolution version via Google Drive: http://bit.ly/COCO-GAN-full

References in corpus (17)

• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Pixel Recurrent Neural Networks
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Which Training Methods for GANs do actually Converge?
• A Learned Representation For Artistic Style
• Adversarially Learned Inference
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
• Modulating early visual processing by language
• IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis
• Painting Outside the Box: Image Outpainting with GANs
• HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentation

Cited by in corpus (2)

• Divide-and-Conquer Adversarial Learning for High-Resolution Image and Video Enhancement
• What it Thinks is Important is Important: Robustness Transfers through Input Gradients