Expectation-Maximization Attention Networks for Semantic Segmentation

Abstract

Self-attention mechanism has been widely used for various tasks. It is designed to compute the representation of each position by a weighted sum of the features at all positions. Thus, it can capture long-range relations for computer vision tasks. However, it is computationally consuming. Since the attention maps are computed w.r.t all other positions. In this paper, we formulate the attention mechanism into an expectation-maximization manner and iteratively estimate a much more compact set of bases upon which the attention maps are computed. By a weighted summation upon these bases, the resulting representation is low-rank and deprecates noisy information from the input. The proposed Expectation-Maximization Attention (EMA) module is robust to the variance of input and is also friendly in memory and computation. Moreover, we set up the bases maintenance and normalization methods to stabilize its training procedure. We conduct extensive experiments on popular semantic segmentation benchmarks including PASCAL VOC, PASCAL Context and COCO Stuff, on which we set new records.

arXiv commentIn Proceedings of International Conference in Computer Vision (ICCV), 2019. Oral

References in corpus (2)

• Rethinking Atrous Convolution for Semantic Image Segmentation
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Cited by in corpus (13)

• ABCNet: Attentive Bilateral Contextual Network for Efficient Semantic Segmentation of Fine-Resolution Remote Sensing Images
• Hierarchical Multi-Scale Attention for Semantic Segmentation
• K-Net: Towards Unified Image Segmentation
• Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
• Uncertainty-Aware Consistency Regularization for Cross-Domain Semantic Segmentation
• GFF: Gated Fully Fusion for Semantic Segmentation
• Prototype Mixture Models for Few-shot Semantic Segmentation
• PointFlow: Flowing Semantics Through Points for Aerial Image Segmentation
• BoundarySqueeze: Image Segmentation as Boundary Squeezing
• Deep Convolutional Neural Networks with Spatial Regularization, Volume and Star-shape Priori for Image Segmentation
• SOGNet: Scene Overlap Graph Network for Panoptic Segmentation
• One-Shot Object Affordance Detection in the Wild
• Hierarchical Pyramid Representations for Semantic Segmentation