Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks

Abstract

The use of RGB-D information for salient object detection has been extensively explored in recent years. However, relatively few efforts have been put towards modeling salient object detection in real-world human activity scenes with RGBD. In this work, we fill the gap by making the following contributions to RGB-D salient object detection. (1) We carefully collect a new SIP (salient person) dataset, which consists of ~1K high-resolution images that cover diverse real-world scenes from various viewpoints, poses, occlusions, illuminations, and backgrounds. (2) We conduct a large-scale (and, so far, the most comprehensive) benchmark comparing contemporary methods, which has long been missing in the field and can serve as a baseline for future research. We systematically summarize 32 popular models and evaluate 18 parts of 32 models on seven datasets containing a total of about 97K images. (3) We propose a simple general architecture, called Deep Depth-Depurator Network (D3Net). It consists of a depth depurator unit (DDU) and a three-stream feature learning module (FLM), which performs low-quality depth map filtering and cross-modal feature learning respectively. These components form a nested structure and are elaborately designed to be learned jointly. D3Net exceeds the performance of any prior contenders across all five metrics under consideration, thus serving as a strong model to advance research in this field. We also demonstrate that D3Net can be used to efficiently extract salient object masks from real scenes, enabling effective background changing application with a speed of 65fps on a single GPU. All the saliency maps, our new SIP dataset, the D3Net model, and the evaluation tools are publicly available at https://github.com/DengPingFan/D3NetBenchmark.

arXiv commentAccepted in TNNLS20. 15 pages, 12 figures. Code: https://github.com/DengPingFan/D3NetBenchmark

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Res2Net: A New Multi-scale Backbone Architecture
• Salient Object Detection: A Benchmark
• Deeply supervised salient object detection with short connections
• Salient Object Detection: A Survey
• Review of Visual Saliency Detection with Comprehensive Information
• RGBD Salient Object Detection via Deep Fusion
• DISC: Deep Image Saliency Computing via Progressive Representation Learning
• Co-saliency Detection for RGBD Images Based on Multi-constraint Feature Matching and Cross Label Propagation
• Salient Object Detection with Lossless Feature Reflection and Weighted Structural Loss
• Selectivity or Invariance: Boundary-aware Salient Object Detection
• Richer and Deeper Supervision Network for Salient Object Detection
• Saliency Integration: An Arbitrator Model
• Deep Reasoning with Multi-Scale Context for Salient Object Detection

Cited by in corpus (70)

• Concealed Object Detection
• SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection
• RGB-D Salient Object Detection: A Survey
• RANet: Ranking Attention Network for Fast Video Object Segmentation
• Multi-interactive Dual-decoder for RGB-thermal Salient Object Detection
• DPANet: Depth Potentiality-Aware Gated Attention Network for RGB-D Salient Object Detection
• CAVER: Cross-Modal View-Mixed Transformer for Bi-Modal Salient Object Detection
• CIR-Net: Cross-modality Interaction and Refinement for RGB-D Salient Object Detection
• Deep Hough Transform for Semantic Line Detection
• Bilateral Attention Network for RGB-D Salient Object Detection
• Video Polyp Segmentation: A Deep Learning Perspective
• MobileSal: Extremely Efficient RGB-D Salient Object Detection
• Bifurcated backbone strategy for RGB-D salient object detection
• TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Network
• HiDAnet: RGB-D Salient Object Detection via Hierarchical Depth Awareness
• HRTransNet: HRFormer-Driven Two-Modality Salient Object Detection
• Re-thinking Co-Salient Object Detection
• Advances in Deep Concealed Scene Understanding
• Depth Quality Aware Salient Object Detection
• Light Field Salient Object Detection: A Review and Benchmark
• Data-Level Recombination and Lightweight Fusion Scheme for RGB-D Salient Object Detection
• EGNet:Edge Guidance Network for Salient Object Detection
• Efficient and Accurate Arbitrary-Shaped Text Detection with Pixel Aggregation Network
• Middle-level Fusion for Lightweight RGB-D Salient Object Detection
• VST++: Efficient and Stronger Visual Saliency Transformer
• Salient Objects in Clutter
• RGB-D Salient Object Detection with Ubiquitous Target Awareness
• Suppress and Balance: A Simple Gated Network for Salient Object Detection
• UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders
• Contrast-weighted Dictionary Learning Based Saliency Detection for Remote Sensing Images
• Joint Learning of Salient Object Detection, Depth Estimation and Contour Extraction
• JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detection
• Boosting RGB-D Saliency Detection by Leveraging Unlabeled RGB Images
• Siamese Network for RGB-D Salient Object Detection and Beyond
• Multi-Scale Iterative Refinement Network for RGB-D Salient Object Detection
• A Single Stream Network for Robust and Real-time RGB-D Salient Object Detection
• Deep Fourier-embedded Network for RGB and Thermal Salient Object Detection
• View-aware Salient Object Detection for 360° Omnidirectional Image
• ViDSOD-100: A New Dataset and a Baseline Model for RGB-D Video Salient Object Detection
• Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detection
• Accurate RGB-D Salient Object Detection via Collaborative Learning
• Learning Synergistic Attention for Light Field Salient Object Detection
• RGB-D Saliency Detection via Cascaded Mutual Information Minimization
• Cross-Modal Weighting Network for RGB-D Salient Object Detection
• Uncertainty Inspired RGB-D Saliency Detection
• Progressively Guided Alternate Refinement Network for RGB-D Salient Object Detection
• Modal-Adaptive Gated Recoding Network for RGB-D Salient Object Detection
• Transformer-based Network for RGB-D Saliency Detection
• Cross-modality Discrepant Interaction Network for RGB-D Salient Object Detection
• Learning Selective Mutual Attention and Contrast for RGB-D Saliency Detection
• Depth Quality-Inspired Feature Manipulation for Efficient RGB-D Salient Object Detection
• BridgeNet: A Joint Learning Network of Depth Map Super-Resolution and Monocular Depth Estimation
• CMA-Net: A Cascaded Mutual Attention Network for Light Field Salient Object Detection
• Multi-Source Fusion and Automatic Predictor Selection for Zero-Shot Video Object Segmentation
• RRNet: Relational Reasoning Network with Parallel Multi-scale Attention for Salient Object Detection in Optical Remote Sensing Images
• LEAF-Mamba: Local Emphatic and Adaptive Fusion State Space Model for RGB-D Salient Object Detection
• Multi-level Cross-modal Interaction Network for RGB-D Salient Object Detection
• Synergistic saliency and depth prediction for RGB-D saliency detection
• A Fixation-based 360° Benchmark Dataset for Salient Object Detection
• BTS-Net: Bi-directional Transfer-and-Selection Network For RGB-D Salient Object Detection
• Is Depth Really Necessary for Salient Object Detection?
• A Unified Structure for Efficient RGB and RGB-D Salient Object Detection
• SHD360: A Benchmark Dataset for Salient Human Detection in 360° Videos
• Densely Deformable Efficient Salient Object Detection Network
• Perception-and-Regulation Network for Salient Object Detection
• Cascade Graph Neural Networks for RGB-D Salient Object Detection
• Visual Boundary Knowledge Translation for Foreground Segmentation
• ACFNet: Adaptively-Cooperative Fusion Network for RGB-D Salient Object Detection
• \emph{cm}SalGAN: RGB-D Salient Object Detection with Cross-View Generative Adversarial Networks
• Scoot: A Perceptual Metric for Facial Sketches