CAVER: Cross-Modal View-Mixed Transformer for Bi-Modal Salient Object Detection

Abstract

Most of the existing bi-modal (RGB-D and RGB-T) salient object detection methods utilize the convolution operation and construct complex interweave fusion structures to achieve cross-modal information integration. The inherent local connectivity of the convolution operation constrains the performance of the convolution-based methods to a ceiling. In this work, we rethink these tasks from the perspective of global information alignment and transformation. Specifically, the proposed \underline{c}ross-mod\underline{a}l \underline{v}iew-mixed transform\underline{er} (CAVER) cascades several cross-modal integration units to construct a top-down transformer-based information propagation path. CAVER treats the multi-scale and multi-modal feature integration as a sequence-to-sequence context propagation and update process built on a novel view-mixed attention mechanism. Besides, considering the quadratic complexity w.r.t. the number of input tokens, we design a parameter-free patch-wise token re-embedding strategy to simplify operations. Extensive experimental results on RGB-D and RGB-T SOD datasets demonstrate that such a simple two-stream encoder-decoder framework can surpass recent state-of-the-art methods when it is equipped with the proposed components. Code and pretrained models will be available at \href{https://github.com/lartpang/CAVER}{the link}.

arXiv commentAccepted by TIP-2023. Add more details and update the weight illustration

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Res2Net: A New Multi-scale Backbone Architecture
• Concealed Object Detection
• Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks
• STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation
• Salient Object Detection: A Survey
• RGBD Salient Object Detection via Deep Fusion
• RGB-D Salient Object Detection: A Survey
• Multi-interactive Dual-decoder for RGB-thermal Salient Object Detection
• Bilateral Attention Network for RGB-D Salient Object Detection
• Bifurcated backbone strategy for RGB-D salient object detection
• TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Network
• Dynamic Feature Integration for Simultaneous Detection of Salient Object, Edge and Skeleton
• Light Field Salient Object Detection: A Review and Benchmark
• RGBT Salient Object Detection: A Large-scale Dataset and Benchmark

Cited by in corpus (4)

• UniRGB-IR: A Unified Framework for Visible-Infrared Semantic Tasks via Adapter Tuning
• Deep Fourier-embedded Network for RGB and Thermal Salient Object Detection
• Efficient Fourier Filtering Network with Contrastive Learning for AAV-based Unaligned Bimodal Salient Object Detection
• LEAF-Mamba: Local Emphatic and Adaptive Fusion State Space Model for RGB-D Salient Object Detection