SeGAN: Segmenting and Generating the Invisible
arXiv:1703.10239
Abstract
Objects often occlude each other in scenes; Inferring their appearance beyond their visible parts plays an important role in scene understanding, depth estimation, object interaction and manipulation. In this paper, we study the challenging problem of completing the appearance of occluded objects. Doing so requires knowing which pixels to paint (segmenting the invisible parts of objects) and what color to paint them (generating the invisible parts). Our proposed novel solution, SeGAN, jointly optimizes for both segmentation and generation of the invisible parts of objects. Our experimental results show that: (a) SeGAN can learn to generate the appearance of the occluded parts of objects; (b) SeGAN outperforms state-of-the-art segmentation baselines for the invisible parts of objects; (c) trained on synthetic photo realistic images, SeGAN can reliably segment natural images; (d) by reasoning about occluder occludee relations, our method can infer depth layering.
arXiv commentAccepted to CVPR18 as spotlight
References in corpus (22)
- Conditional Generative Adversarial Nets
- Deep Residual Learning for Image Recognition
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
- Fully Convolutional Networks for Semantic Segmentation
- Learning to Segment Object Candidates
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
- Learning Deconvolution Network for Semantic Segmentation
- Rich feature hierarchies for accurate object detection and semantic segmentation
- Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation
- Scalable, High-Quality Object Detection
- Semantic Image Segmentation via Deep Parsing Network
- Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
- BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation
- Attention to Scale: Scale-aware Semantic Image Segmentation
- Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
- Instance-aware Semantic Segmentation via Multi-task Network Cascades
- Instance-sensitive Fully Convolutional Networks
- Instance-Level Segmentation for Autonomous Driving with Deep Densely Connected MRFs
- Iterative Instance Segmentation
- Amodal Instance Segmentation
- Amodal Completion and Size Constancy in Natural Scenes
- Semantic Amodal Segmentation