Attention to Scale: Scale-aware Semantic Image Segmentation

Abstract

Incorporating multi-scale features in fully convolutional neural networks (FCNs) has been a key element to achieving state-of-the-art performance on semantic image segmentation. One common way to extract multi-scale features is to feed multiple resized input images to a shared deep network and then merge the resulting features for pixelwise classification. In this work, we propose an attention mechanism that learns to softly weight the multi-scale features at each pixel location. We adapt a state-of-the-art semantic image segmentation model, which we jointly train with multi-scale input images and the attention model. The proposed attention model not only outperforms average- and max-pooling, but allows us to diagnostically visualize the importance of features at different positions and scales. Moreover, we show that adding extra supervision to the output at each scale is essential to achieving excellent performance when merging multi-scale features. We demonstrate the effectiveness of our model with extensive experiments on three challenging datasets, including PASCAL-Person-Part, PASCAL VOC 2012 and a subset of MS-COCO 2014.

14 pages. Accepted to appear at CVPR 2016

References in corpus (3)

• Multiple Object Recognition with Visual Attention
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Semantic Image Segmentation with Task-Specific Edge Detection Using CNNs and a Discriminatively Trained Domain Transform

Cited by in corpus (42)

• Multi-Scale Context Aggregation by Dilated Convolutions
• Hierarchical Multi-Scale Attention for Semantic Segmentation
• Residual Attention Network for Image Classification
• Pyramid Scene Parsing Network
• Light-Weight RefineNet for Real-Time Semantic Segmentation
• HydraPlus-Net: Attentive Deep Features for Pedestrian Analysis
• Learning a Discriminative Feature Network for Semantic Segmentation
• A Method of Rapid Quantification of Patient-Specific Organ Dose for CT Using Coupled Deep-Learning based Multi-Organ Segmentation and GPU-accelerated Monte Carlo Dose Computing
• Gated-Dilated Networks for Lung Nodule Classification in CT scans
• RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation
• PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing
• Real-time Semantic Image Segmentation via Spatial Sparsity
• STFCN: Spatio-Temporal FCN for Semantic Video Segmentation
• Deep Learning Convolutional Networks for Multiphoton Microscopy Vasculature Segmentation
• Unite the People: Closing the Loop Between 3D and 2D Human Representations
• Semantic Image Segmentation with Task-Specific Edge Detection Using CNNs and a Discriminatively Trained Domain Transform
• Thickened 2D Networks for Efficient 3D Medical Image Segmentation
• Iterative Visual Reasoning Beyond Convolutions
• ScaleNet: Guiding Object Proposal Generation in Supermarkets and Beyond
• Decoupled Pyramid Correlation Network for Liver Tumor Segmentation from CT images
• Joint Multi-Person Pose Estimation and Semantic Part Segmentation
• Dense Recurrent Neural Networks for Scene Labeling
• Weakly and Semi Supervised Human Body Part Parsing via Pose-Guided Knowledge Transfer
• Semantic Object Parsing with Graph LSTM
• Semantic Video Segmentation: A Review on Recent Approaches
• Error Correction for Dense Semantic Image Labeling
• EfficientHRNet: Efficient Scaling for Lightweight High-Resolution Multi-Person Pose Estimation
• SeGAN: Segmenting and Generating the Invisible
• Mixed context networks for semantic segmentation
• Sparsely Grouped Multi-task Generative Adversarial Networks for Facial Attribute Manipulation
• Bipartite Conditional Random Fields for Panoptic Segmentation
• SurfConv: Bridging 3D and 2D Convolution for RGBD Images
• Ordinal Depth Supervision for 3D Human Pose Estimation
• Low-Latency Video Semantic Segmentation
• Contrast-Oriented Deep Neural Networks for Salient Object Detection
• Instance-Level Salient Object Segmentation
• Dynamic Video Segmentation Network
• CDGNet: Class Distribution Guided Network for Human Parsing
• Beyond Planar Symmetry: Modeling human perception of reflection and rotation symmetries in the wild
• Scene Parsing via Dense Recurrent Neural Networks with Attentional Selection
• Efficient Modelling Across Time of Human Actions and Interactions
• Generating Attention from Classifier Activations for Fine-grained Recognition