Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation

Abstract

Convolution exploits locality for efficiency at a cost of missing long range context. Self-attention has been adopted to augment CNNs with non-local interactions. Recent works prove it possible to stack self-attention layers to obtain a fully attentional network by restricting the attention to a local region. In this paper, we attempt to remove this constraint by factorizing 2D self-attention into two 1D self-attentions. This reduces computation complexity and allows performing attention within a larger or even global region. In companion, we also propose a position-sensitive self-attention design. Combining both yields our position-sensitive axial-attention layer, a novel building block that one could stack to form axial-attention models for image classification and dense prediction. We demonstrate the effectiveness of our model on four large-scale datasets. In particular, our model outperforms all existing stand-alone self-attention models on ImageNet. Our Axial-DeepLab improves 2.8% PQ over bottom-up state-of-the-art on COCO test-dev. This previous state-of-the-art is attained by our small variant that is 3.8x parameter-efficient and 27x computation-efficient. Axial-DeepLab also achieves state-of-the-art results on Mapillary Vistas and Cityscapes.

arXiv commentECCV 2020 camera-ready

References in corpus (24)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Neural Architecture Search with Reinforcement Learning
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Self-Attention Generative Adversarial Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Benchmark Analysis of Representative Deep Neural Network Architectures
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• Making Convolutional Networks Shift-Invariant Again
• Lookahead Optimizer: k steps forward, 1 step back
• Axial Attention in Multidimensional Transformers
• Deep High-Resolution Representation Learning for Visual Recognition
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• Image Transformer
• DeeperLab: Single-Shot Image Parser
• Learning to Fuse Things and Stuff
• Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation
• GFF: Gated Fully Fusion for Semantic Segmentation
• PolyTransform: Deep Polygon Transformer for Instance Segmentation
• SpatialFlow: Bridging All Tasks for Panoptic Segmentation

Cited by in corpus (15)

• Transformers in Vision: A Survey
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Coordinate Attention for Efficient Mobile Network Design
• DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution
• Attention-Based Transformers for Instance Segmentation of Cells in Microstructures
• Video Instance Segmentation using Inter-Frame Communication Transformers
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• An Attention Free Transformer
• Global Self-Attention Networks for Image Recognition
• SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobile
• Instance and Panoptic Segmentation Using Conditional Convolutions
• Neural Architecture Search for Lightweight Non-Local Networks
• Bipartite Conditional Random Fields for Panoptic Segmentation
• HATNet: An End-to-End Holistic Attention Network for Diagnosis of Breast Biopsy Images
• Dense Prediction with Attentive Feature Aggregation