OCNet: Object Context Network for Scene Parsing

Abstract

In this paper, we address the semantic segmentation task with a new context aggregation scheme named \emph{object context}, which focuses on enhancing the role of object information. Motivated by the fact that the category of each pixel is inherited from the object it belongs to, we define the object context for each pixel as the set of pixels that belong to the same category as the given pixel in the image. We use a binary relation matrix to represent the relationship between all pixels, where the value one indicates the two selected pixels belong to the same category and zero otherwise. We propose to use a dense relation matrix to serve as a surrogate for the binary relation matrix. The dense relation matrix is capable to emphasize the contribution of object information as the relation scores tend to be larger on the object pixels than the other pixels. Considering that the dense relation matrix estimation requires quadratic computation overhead and memory consumption w.r.t. the input size, we propose an efficient interlaced sparse self-attention scheme to model the dense relations between any two of all pixels via the combination of two sparse relation matrices. To capture richer context information, we further combine our interlaced sparse self-attention scheme with the conventional multi-scale context schemes including pyramid pooling~\citep{zhao2017pyramid} and atrous spatial pyramid pooling~\citep{chen2018deeplab}. We empirically show the advantages of our approach with competitive performances on five challenging benchmarks including: Cityscapes, ADE20K, LIP, PASCAL-Context and COCO-Stuff

arXiv commentAccepted at IJCV 2021

References in corpus (21)

• Rethinking Atrous Convolution for Semantic Image Segmentation
• A Structured Self-attentive Sentence Embedding
• ParseNet: Looking Wider to See Better
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• High-Resolution Representations for Labeling Pixels and Regions
• Generating Long Sequences with Sparse Transformers
• UNet++: A Nested U-Net Architecture for Medical Image Segmentation
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Context Encoding for Semantic Segmentation
• Compact Generalized Non-local Network
• Learning a Discriminative Feature Network for Semantic Segmentation
• Efficient Attention: Attention with Linear Complexities
• High-performance Semantic Segmentation Using Very Deep Fully Convolutional Networks
• Asymmetric Non-local Neural Networks for Semantic Segmentation
• Devil in the Details: Towards Accurate Single and Multiple Human Parsing
• Look into Person: Self-supervised Structure-sensitive Learning and A New Benchmark for Human Parsing
• Grid Saliency for Context Explanations of Semantic Segmentation
• IGCV$2$: Interleaved Structured Sparse Convolutional Neural Networks
• ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors
• Adaptive Affinity Fields for Semantic Segmentation
• Tree-structured Kronecker Convolutional Network for Semantic Segmentation

Cited by in corpus (123)

• A Survey on Visual Transformer
• Attention Mechanisms in Computer Vision: A Survey
• SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
• Multi-Task Learning for Dense Prediction Tasks: A Survey
• Twins: Revisiting the Design of Spatial Attention in Vision Transformers
• Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation
• Deep High-Resolution Representation Learning for Visual Recognition
• CCNet: Criss-Cross Attention for Semantic Segmentation
• Hierarchical Multi-Scale Attention for Semantic Segmentation
• Multi-stage Attention ResU-Net for Semantic Segmentation of Fine-Resolution Remote Sensing Images
• MAP-Net: Multi Attending Path Neural Network for Building Footprint Extraction from Remote Sensed Imagery
• Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes
• An Empirical Study of Remote Sensing Pretraining
• GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond
• Dual Attention Network for Scene Segmentation
• Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
• RGB-T Semantic Segmentation with Location, Activation, and Sharpening
• MACU-Net for Semantic Segmentation of Fine-Resolution Remotely Sensed Images
• HRFormer: High-Resolution Transformer for Dense Prediction
• Pre-Trained Image Processing Transformer
• Dual Graph Convolutional Network for Semantic Segmentation
• BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation
• Polarization-driven Semantic Segmentation via Efficient Attention-bridged Fusion
• An Empirical Study of Spatial Attention Mechanisms in Deep Networks
• Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
• CGNet: A Light-weight Context Guided Network for Semantic Segmentation
• Asymmetric Non-local Neural Networks for Semantic Segmentation
• Weak Supervision for Generating Pixel-Level Annotations in Scene Text Segmentation
• Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision
• Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation
• Interlaced Sparse Self-Attention for Semantic Segmentation
• Toward Interpretable Sleep Stage Classification Using Cross-Modal Transformers
• Is Attention Better Than Matrix Decomposition?
• Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation
• Video Semantic Segmentation with Distortion-Aware Feature Correction
• EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation
• AGPCNet: Attention-Guided Pyramid Context Networks for Infrared Small Target Detection
• Global Aggregation then Local Distribution in Fully Convolutional Networks
• ACFNet: Attentional Class Feature Network for Semantic Segmentation
• FoodSAM: Any Food Segmentation
• MiniSeg: An Extremely Minimum Network for Efficient COVID-19 Segmentation
• Strip Pooling: Rethinking Spatial Pooling for Scene Parsing
• Attention-guided Chained Context Aggregation for Semantic Segmentation
• Boosting Night-time Scene Parsing with Learnable Frequency
• Relation-Aware Global Attention for Person Re-identification
• GFF: Gated Fully Fusion for Semantic Segmentation
• SA-Net: Shuffle Attention for Deep Convolutional Neural Networks
• Progressive Sparse Local Attention for Video object detection
• Segmenting Transparent Object in the Wild with Transformer
• Deep Multiphase Level Set for Scene Parsing
• Birds of A Feather Flock Together: Category-Divergence Guidance for Domain Adaptive Segmentation
• Adaptive Prototype Learning and Allocation for Few-Shot Segmentation
• Global Aggregation then Local Distribution for Scene Parsing
• Joint Semantic Segmentation and Boundary Detection using Iterative Pyramid Contexts
• Beyond Human Parts: Dual Part-Aligned Representations for Person Re-Identification
• Graph-Segmenter: Graph Transformer with Boundary-aware Attention for Semantic Segmentation
• Class-wise Dynamic Graph Convolution for Semantic Segmentation
• Semantic Flow for Fast and Accurate Scene Parsing
• Structured Knowledge Distillation for Dense Prediction
• Hybrid Multiple Attention Network for Semantic Segmentation in Aerial Images
• SegFix: Model-Agnostic Boundary Refinement for Segmentation
• Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation
• Disentangled Non-Local Neural Networks
• Segmenting Transparent Objects in the Wild
• Saliency Guided Self-attention Network for Weakly and Semi-supervised Semantic Segmentation
• PointFlow: Flowing Semantics Through Points for Aerial Image Segmentation
• Bidirectional Graph Reasoning Network for Panoptic Segmentation
• Data Augmentation for Skin Lesion using Self-Attention based Progressive Generative Adversarial Network
• FakeMix Augmentation Improves Transparent Object Detection
• AlignSeg: Feature-Aligned Segmentation Networks
• Efficient Coarse-to-Fine Non-Local Module for the Detection of Small Objects
• Unifying Training and Inference for Panoptic Segmentation
• Global and Local Features through Gaussian Mixture Models on Image Semantic Segmentation
• Multiple Instance Segmentation in Brachial Plexus Ultrasound Image Using BPMSegNet
• Feature Pyramid Transformer
• DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation
• Prototype-based Incremental Few-Shot Semantic Segmentation
• Auxiliary Learning for Deep Multi-task Learning
• A Spacecraft Dataset for Detection, Segmentation and Parts Recognition
• CaseNet: Content-Adaptive Scale Interaction Networks for Scene Parsing
• DEAL: Difficulty-aware Active Learning for Semantic Segmentation
• 3D-to-2D Distillation for Indoor Scene Parsing
• Centralized Information Interaction for Salient Object Detection
• AttaNet: Attention-Augmented Network for Fast and Accurate Scene Parsing
• Fast Video Salient Object Detection via Spatiotemporal Knowledge Distillation
• A Unified Efficient Pyramid Transformer for Semantic Segmentation
• FlatteNet: A Simple Versatile Framework for Dense Pixelwise Prediction
• Distance Guided Channel Weighting for Semantic Segmentation
• Semi-Global Shape-aware Network
• Sub-word Level Lip Reading With Visual Attention
• Consensus Feature Network for Scene Parsing
• Local block-wise self attention for normal organ segmentation
• Context-aware Padding for Semantic Segmentation
• Representative Graph Neural Network
• Adaptive Context Encoding Module for Semantic Segmentation
• FixNorm: Dissecting Weight Decay for Training Deep Neural Networks
• Cerberus Transformer: Joint Semantic, Affordance and Attribute Parsing
• EADNet: Efficient Asymmetric Dilated Network for Semantic Segmentation
• AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training
• CDGNet: Class Distribution Guided Network for Human Parsing
• SkyScapes -- Fine-Grained Semantic Understanding of Aerial Scenes
• Capturing Omni-Range Context for Omnidirectional Segmentation
• Context Prior for Scene Segmentation
• DCANet: Dense Context-Aware Network for Semantic Segmentation
• Adaptive Context Network for Scene Parsing
• Denoised Non-Local Neural Network for Semantic Segmentation
• Global Context Networks
• GSANet: Semantic Segmentation with Global and Selective Attention
• GINet: Graph Interaction Network for Scene Parsing
• Recognizing Object Affordances to Support Scene Reasoning for Manipulation Tasks
• Local Context Attention for Salient Object Segmentation
• Feature boosting with efficient attention for scene parsing
• 3D Guided Weakly Supervised Semantic Segmentation
• Shift-and-Balance Attention
• ARM: A Confidence-Based Adversarial Reweighting Module for Coarse Semantic Segmentation
• Heterogeneous Grid Convolution for Adaptive, Efficient, and Controllable Computation
• All You Need is a Second Look: Towards Arbitrary-Shaped Text Detection
• BiCANet: Bi-directional Contextual Aggregating Network for Image Semantic Segmentation
• Deep Grouping Model for Unified Perceptual Parsing
• HCNet: Hierarchical Context Network for Semantic Segmentation
• Covariance Self-Attention Dual Path UNet for Rectal Tumor Segmentation
• Hierarchical Pyramid Representations for Semantic Segmentation
• TB-Net: A Three-Stream Boundary-Aware Network for Fine-Grained Pavement Disease Segmentation