Hierarchical Multi-Scale Attention for Semantic Segmentation

Abstract

Multi-scale inference is commonly used to improve the results of semantic segmentation. Multiple images scales are passed through a network and then the results are combined with averaging or max pooling. In this work, we present an attention-based approach to combining multi-scale predictions. We show that predictions at certain scales are better at resolving particular failures modes, and that the network learns to favor those scales for such cases in order to generate better predictions. Our attention mechanism is hierarchical, which enables it to be roughly 4x more memory efficient to train than other recent approaches. In addition to enabling faster training, this allows us to train with larger crop sizes which leads to greater model accuracy. We demonstrate the result of our method on two datasets: Cityscapes and Mapillary Vistas. For Cityscapes, which has a large number of weakly labelled images, we also leverage auto-labelling to improve generalization. Using our approach we achieve a new state-of-the-art results in both Mapillary (61.1 IOU val) and Cityscapes (85.1 IOU test).

11 pages, 5 figures

References in corpus (12)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Rethinking Atrous Convolution for Semantic Image Segmentation
• OCNet: Object Context Network for Scene Parsing
• CCNet: Criss-Cross Attention for Semantic Segmentation
• Pyramid Attention Network for Semantic Segmentation
• DeeperLab: Single-Shot Image Parser
• Attention to Scale: Scale-aware Semantic Image Segmentation
• Expectation-Maximization Attention Networks for Semantic Segmentation
• Region Mutual Information Loss for Semantic Segmentation
• Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training
• ACFNet: Attentional Class Feature Network for Semantic Segmentation
• Graph-Based Global Reasoning Networks

Cited by in corpus (45)

• Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation
• Deep Dense Multi-scale Network for Snow Removal Using Semantic and Geometric Priors
• Per-Pixel Classification is Not All You Need for Semantic Segmentation
• Polarized Self-Attention: Towards High-quality Pixel-wise Regression
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• Masked-attention Mask Transformer for Universal Image Segmentation
• PaddleSeg: A High-Efficient Development Toolkit for Image Segmentation
• CitySurfaces: City-Scale Semantic Segmentation of Sidewalk Materials
• MGNet: Monocular Geometric Scene Understanding for Autonomous Driving
• Do We Really Need Dice? The Hidden Region-Size Biases of Segmentation Losses
• A Comprehensive Review of Modern Object Segmentation Approaches
• CCVS: Context-aware Controllable Video Synthesis
• Learning to Downsample for Segmentation of Ultra-High Resolution Images
• Adaptive Prototype Learning and Allocation for Few-Shot Segmentation
• Mask Guided Attention For Fine-Grained Patchy Image Classification
• Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation
• A Survey on Deep Learning Methods for Semantic Image Segmentation in Real-Time
• Hierarchical Random Walker Segmentation for Large Volumetric Biomedical Images
• Comprehensive Multi-Modal Interactions for Referring Image Segmentation
• Multi-Scale Feature Aggregation by Cross-Scale Pixel-to-Region Relation Operation for Semantic Segmentation
• Toward Compact Deep Neural Networks via Energy-Aware Pruning
• ATSal: An Attention Based Architecture for Saliency Prediction in 360 Videos
• Global and Local Features through Gaussian Mixture Models on Image Semantic Segmentation
• Accelerating Neural Network Training with Distributed Asynchronous and Selective Optimization (DASO)
• EfficientSeg: An Efficient Semantic Segmentation Network
• Attentional Feature Fusion
• Multi-modal anticipation of stochastic trajectories in a dynamic environment with Conditional Variational Autoencoders
• Empirical Study of Multi-Task Hourglass Model for Semantic Segmentation Task
• Improving Road Segmentation in Challenging Domains Using Similar Place Priors
• Learning to Segment Human Body Parts with Synthetically Trained Deep Convolutional Networks
• Rapid quantification of COVID-19 pneumonia burden from computed tomography with convolutional LSTM networks
• Boosting Image Outpainting with Semantic Layout Prediction
• ClimateGAN: Raising Climate Change Awareness by Generating Images of Floods
• RaidaR: A Rich Annotated Image Dataset of Rainy Street Scenes
• ICDAR 2021 Competition on Components Segmentation Task of Document Photos
• Monocular Spherical Depth Estimation with Explicitly Connected Weak Layout Cues
• Quantifying urban streetscapes with deep learning: focus on aesthetic evaluation
• Dense Prediction with Attentive Feature Aggregation
• Bidirectional Multi-scale Attention Networks for Semantic Segmentation of Oblique UAV Imagery
• Perceptual Consistency in Video Segmentation
• MetaPix: Domain Transfer for Semantic Segmentation by Meta Pixel Weighting
• Semantic Segmentation for Urban-Scene Images
• 3rd Place Solution for Short-video Face Parsing Challenge
• TensorX: Extensible API for Neural Network Model Design and Deployment
• CASPIANET++: A Multidimensional Channel-Spatial Asymmetric Attention Network with Noisy Student Curriculum Learning Paradigm for Brain Tumor Segmentation