CCNet: Criss-Cross Attention for Semantic Segmentation

Abstract

Contextual information is vital in visual understanding problems, such as semantic segmentation and object detection. We propose a Criss-Cross Network (CCNet) for obtaining full-image contextual information in a very effective and efficient way. Concretely, for each pixel, a novel criss-cross attention module harvests the contextual information of all the pixels on its criss-cross path. By taking a further recurrent operation, each pixel can finally capture the full-image dependencies. Besides, a category consistent loss is proposed to enforce the criss-cross attention module to produce more discriminative features. Overall, CCNet is with the following merits: 1) GPU memory friendly. Compared with the non-local block, the proposed recurrent criss-cross attention module requires 11x less GPU memory usage. 2) High computational efficiency. The recurrent criss-cross attention significantly reduces FLOPs by about 85% of the non-local block. 3) The state-of-the-art performance. We conduct extensive experiments on semantic segmentation benchmarks including Cityscapes, ADE20K, human parsing benchmark LIP, instance segmentation benchmark COCO, video segmentation benchmark CamVid. In particular, our CCNet achieves the mIoU scores of 81.9%, 45.76% and 55.47% on the Cityscapes test set, the ADE20K validation set and the LIP validation set respectively, which are the new state-of-the-art results. The source codes are available at \url{https://github.com/speedinghzl/CCNet}.

arXiv commentIEEE TPAMI 2020 & ICCV 2019

References in corpus (13)

• Semi-Supervised Classification with Graph Convolutional Networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Neural Message Passing for Quantum Chemistry
• Conditional Random Fields as Recurrent Neural Networks
• Deep Convolutional Networks on Graph-Structured Data
• ParseNet: Looking Wider to See Better
• OCNet: Object Context Network for Scene Parsing
• Semantic Instance Segmentation with a Discriminative Loss Function
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
• Long Short-Term Memory-Networks for Machine Reading
• SPGNet: Semantic Prediction Guidance for Scene Parsing
• Adaptive Affinity Fields for Semantic Segmentation

Cited by in corpus (24)

• Hierarchical Multi-Scale Attention for Semantic Segmentation
• Coordinate Attention for Efficient Mobile Network Design
• GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond
• An Empirical Study of Spatial Attention Mechanisms in Deep Networks
• XLSor: A Robust and Accurate Lung Segmentor on Chest X-Rays Using Criss-Cross Attention and Customized Radiorealistic Abnormalities Generation
• A Single-Shot Arbitrarily-Shaped Text Detector based on Context Attended Multi-Task Learning
• Strip Pooling: Rethinking Spatial Pooling for Scene Parsing
• AugFPN: Improving Multi-scale Feature Learning for Object Detection
• Attention-guided Chained Context Aggregation for Semantic Segmentation
• Kronecker Attention Networks
• GFF: Gated Fully Fusion for Semantic Segmentation
• A Survey of Simultaneous Localization and Mapping with an Envision in 6G Wireless Networks
• Assessing YOLACT++ for real time and robust instance segmentation of medical instruments in endoscopic procedures
• FabGPT: An Efficient Large Multimodal Model for Complex Wafer Defect Knowledge Queries
• Class-wise Dynamic Graph Convolution for Semantic Segmentation
• Efficient Folded Attention for 3D Medical Image Reconstruction and Segmentation
• GPS-Net: Graph Property Sensing Network for Scene Graph Generation
• Local block-wise self attention for normal organ segmentation
• Linear Context Transform Block
• High-Order Paired-ASPP Networks for Semantic Segmenation
• Adaptive Context Network for Scene Parsing
• Global Context Networks
• A Spectral Nonlocal Block for Neural Networks
• Image Super-Resolution via Attention based Back Projection Networks