Interlaced Sparse Self-Attention for Semantic Segmentation

Abstract

In this paper, we present a so-called interlaced sparse self-attention approach to improve the efficiency of the \emph{self-attention} mechanism for semantic segmentation. The main idea is that we factorize the dense affinity matrix as the product of two sparse affinity matrices. There are two successive attention modules each estimating a sparse affinity matrix. The first attention module is used to estimate the affinities within a subset of positions that have long spatial interval distances and the second attention module is used to estimate the affinities within a subset of positions that have short spatial interval distances. These two attention modules are designed so that each position is able to receive the information from all the other positions. In contrast to the original self-attention module, our approach decreases the computation and memory complexity substantially especially when processing high-resolution feature maps. We empirically verify the effectiveness of our approach on six challenging semantic segmentation benchmarks.

References in corpus (8)

• Rethinking Atrous Convolution for Semantic Image Segmentation
• OCNet: Object Context Network for Scene Parsing
• Generating Long Sequences with Sparse Transformers
• DeeperLab: Single-Shot Image Parser
• Compact Generalized Non-local Network
• Devil in the Details: Towards Accurate Single and Multiple Human Parsing
• IGCV$2$: Interleaved Structured Sparse Convolutional Neural Networks
• Tree-structured Kronecker Convolutional Network for Semantic Segmentation

Cited by in corpus (23)

• A Survey on Visual Transformer
• Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation
• Deep High-Resolution Representation Learning for Visual Recognition
• HRFormer: High-Resolution Transformer for Dense Prediction
• Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer
• IENet: Interacting Embranchment One Stage Anchor Free Detector for Orientation Aerial Object Detection
• An Attention Free Transformer
• ORDNet: Capturing Omni-Range Dependencies for Scene Parsing
• Beyond Human Parts: Dual Part-Aligned Representations for Person Re-Identification
• Hybrid Multiple Attention Network for Semantic Segmentation in Aerial Images
• SegFix: Model-Agnostic Boundary Refinement for Segmentation
• Hierarchical Opacity Propagation for Image Matting
• Saliency Guided Self-attention Network for Weakly and Semi-supervised Semantic Segmentation
• LRNNet: A Light-Weighted Network with Efficient Reduced Non-Local Operation for Real-Time Semantic Segmentation
• Attentive Normalization for Conditional Image Generation
• CTNet: Context-based Tandem Network for Semantic Segmentation
• Salient Positions based Attention Network for Image Classification
• Few-shot Medical Image Segmentation using a Global Correlation Network with Discriminative Embedding
• Representative Graph Neural Network
• CDGNet: Class Distribution Guided Network for Human Parsing
• Retrieval and Localization with Observation Constraints
• Capturing Omni-Range Context for Omnidirectional Segmentation
• DAN: A Deformation-Aware Network for Consecutive Biomedical Image Interpolation