Polarization-driven Semantic Segmentation via Efficient Attention-bridged Fusion

Abstract

Semantic Segmentation (SS) is promising for outdoor scene perception in safety-critical applications like autonomous vehicles, assisted navigation and so on. However, traditional SS is primarily based on RGB images, which limits the reliability of SS in complex outdoor scenes, where RGB images lack necessary information dimensions to fully perceive unconstrained environments. As preliminary investigation, we examine SS in an unexpected obstacle detection scenario, which demonstrates the necessity of multimodal fusion. Thereby, in this work, we present EAFNet, an Efficient Attention-bridged Fusion Network to exploit complementary information coming from different optical sensors. Specifically, we incorporate polarization sensing to obtain supplementary information, considering its optical characteristics for robust representation of diverse materials. By using a single-shot polarization sensor, we build the first RGB-P dataset which consists of 394 annotated pixel-aligned RGB-Polarization images. A comprehensive variety of experiments shows the effectiveness of EAFNet to fuse polarization and RGB information, as well as the flexibility to be adapted to other sensor combination scenarios.

arXiv commentAccepted by Optics Express. 18 pages, 16 figures, 3 tables, 9 equations. Code will be made publicly available at https://github.com/Katexiang/EAFNet

References in corpus (5)

• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• OCNet: Object Context Network for Scene Parsing
• Segmenting Transparent Objects in the Wild
• An end-to-end CNN framework for polarimetric vision tasks based on polarization-parameter-constructing network
• A Multimodal Vision Sensor for Autonomous Driving

Cited by in corpus (8)

• RGB-D And Thermal Sensor Fusion: A Systematic Literature Review
• A Comprehensive Review of Modern Object Segmentation Approaches
• Panoramic Depth Estimation via Supervised and Unsupervised Learning in Indoor Scenes
• ShareCMP: Polarization-Aware RGB-P Semantic Segmentation
• Pola4All: survey of polarimetric applications and an open-source toolkit to analyze polarization
• CSFNet: A Cosine Similarity Fusion Network for Real-Time RGB-X Semantic Segmentation of Driving Scenes
• Capturing Omni-Range Context for Omnidirectional Segmentation
• Trans4Trans: Efficient Transformer for Transparent Object Segmentation to Help Visually Impaired People Navigate in the Real World