Gated-SCNN: Gated Shape CNNs for Semantic Segmentation

Abstract

Current state-of-the-art methods for image segmentation form a dense image representation where the color, shape and texture information are all processed together inside a deep CNN. This however may not be ideal as they contain very different type of information relevant for recognition. Here, we propose a new two-stream CNN architecture for semantic segmentation that explicitly wires shape information as a separate processing branch, i.e. shape stream, that processes information in parallel to the classical stream. Key to this architecture is a new type of gates that connect the intermediate layers of the two streams. Specifically, we use the higher-level activations in the classical stream to gate the lower-level activations in the shape stream, effectively removing noise and helping the shape stream to only focus on processing the relevant boundary-related information. This enables us to use a very shallow architecture for the shape stream that operates on the image-level resolution. Our experiments show that this leads to a highly effective architecture that produces sharper predictions around object boundaries and significantly boosts performance on thinner and smaller objects. Our method achieves state-of-the-art performance on the Cityscapes benchmark, in terms of both mask (mIoU) and boundary (F-score) quality, improving by 2% and 4% over strong baselines.

Project Website: https://nv-tlabs.github.io/GSCNN/

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Categorical Reparameterization with Gumbel-Softmax
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• Conditional Random Fields as Recurrent Neural Networks
• Conditional Image Generation with PixelCNN Decoders
• Densely Connected Convolutional Networks
• Wide Residual Networks
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• Fully Connected Deep Structured Networks
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• Tree-structured Kronecker Convolutional Network for Semantic Segmentation

Cited by in corpus (11)

• Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey
• Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation
• Structured Consistency Loss for semi-supervised semantic segmentation
• Semantic Edge Detection with Diverse Deep Supervision
• Searching for Winograd-aware Quantized Networks
• Deep Convolutional Neural Networks with Spatial Regularization, Volume and Star-shape Priori for Image Segmentation
• A Gated and Bifurcated Stacked U-Net Module for Document Image Dewarping
• Boundary Knowledge Translation based Reference Semantic Segmentation
• Gated Texture CNN for Efficient and Configurable Image Denoising
• PDFNet: Pointwise Dense Flow Network for Urban-Scene Segmentation
• RectiNet-v2: A stacked network architecture for document image dewarping