ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing

Abstract

We address the problem of finding realistic geometric corrections to a foreground object such that it appears natural when composited into a background image. To achieve this, we propose a novel Generative Adversarial Network (GAN) architecture that utilizes Spatial Transformer Networks (STNs) as the generator, which we call Spatial Transformer GANs (ST-GANs). ST-GANs seek image realism by operating in the geometric warp parameter space. In particular, we exploit an iterative STN warping scheme and propose a sequential training strategy that achieves better results compared to naive training of a single generator. One of the key advantages of ST-GAN is its applicability to high-resolution images indirectly since the predicted warp parameters are transferable between reference frames. We demonstrate our approach in two applications: (1) visualizing how indoor furniture (e.g. from product images) might be perceived in a room, (2) hallucinating how accessories like glasses would look when matched with real portraits.

arXiv commentAccepted to CVPR 2018 (website & code: https://chenhsuanlin.bitbucket.io/spatial-transformer-GAN/)

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Improved Training of Wasserstein GANs
• Energy-based Generative Adversarial Network
• Generating Videos with Scene Dynamics
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Deformable Convolutional Networks
• Unsupervised Learning of Depth and Ego-Motion from Video
• LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
• Deep Image Homography Estimation
• Least Squares Generative Adversarial Networks
• Deep Learning Face Attributes in the Wild
• Semantic Facial Expression Editing using Autoencoded Flow
• Semantic Scene Completion from a Single Depth Image
• Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
• WarpNet: Weakly Supervised Matching for Single-view Reconstruction
• Deep Image Harmonization
• Learning a Discriminative Model for the Perception of Realism in Composite Images
• Inverse Compositional Spatial Transformer Networks
• DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation
• A-Fast-RCNN: Hard Positive Generation via Adversary for Object Detection

Cited by in corpus (12)

• Towards Realistic 3D Embedding via View Alignment
• Explicit Disentanglement of Appearance and Perspective in Generative Models
• OPA: Object Placement Assessment Dataset
• Group Equivariant Generative Adversarial Networks
• Instance-level Facial Attributes Transfer with Geometry-Aware Flow
• WarpGAN: Automatic Caricature Generation
• GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving
• Learning Segmentation Masks with the Independence Prior
• A Layer-Based Sequential Framework for Scene Generation with GANs
• Inserting Videos into Videos
• Unconstrained Foreground Object Search
• Continuous learning of face attribute synthesis