Cross-view Semantic Segmentation for Sensing Surroundings
arXiv:1906.03560 · doi:10.1109/LRA.2020.3004325
Abstract
Sensing surroundings plays a crucial role in human spatial perception, as it extracts the spatial configuration of objects as well as the free space from the observations. To facilitate the robot perception with such a surrounding sensing capability, we introduce a novel visual task called Cross-view Semantic Segmentation as well as a framework named View Parsing Network (VPN) to address it. In the cross-view semantic segmentation task, the agent is trained to parse the first-view observations into a top-down-view semantic map indicating the spatial location of all the objects at pixel-level. The main issue of this task is that we lack the real-world annotations of top-down-view data. To mitigate this, we train the VPN in 3D graphics environment and utilize the domain adaptation technique to transfer it to handle real-world data. We evaluate our VPN on both synthetic and real-world agents. The experimental results show that our model can effectively make use of the information from different views and multi-modalities to understanding spatial information. Our further experiment on a LoCoBot robot shows that our model enables the surrounding sensing capability from 2D image input. Code and demo videos can be found at \url{https://view-parsing-network.github.io}.
References in corpus (23)
- Learning Transferable Features with Deep Adaptation Networks
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- High-Resolution Representations for Labeling Pixels and Regions
- Domain Separation Networks
- ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
- Unsupervised Cross-Domain Image Generation
- Matterport3D: Learning from RGB-D Data in Indoor Environments
- CARLA: An Open Urban Driving Simulator
- Building Generalizable Agents with a Realistic and Rich 3D Environment
- SegICP: Integrated Deep Semantic Segmentation and Pose Estimation
- 3D ShapeNets: A Deep Representation for Volumetric Shapes
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- PyRobot: An Open-source Robotics Framework for Research and Benchmarking
- Semantic Scene Completion from a Single Depth Image
- Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images
- Learning Exploration Policies for Navigation
- LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image
- Visual Memory for Robust Path Following
- Situational Fusion of Visual Representation for Visual Navigation
- DSAC - Differentiable RANSAC for Camera Localization
- Generative Adversarial Frontal View to Bird View Synthesis
- Pano2CAD: Room Layout From A Single Panorama Image
- Geometry-Aware Learning of Maps for Camera Localization
Cited by in corpus (22)
- Loss Functions and Metrics in Deep Learning
- Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images
- FISHING Net: Future Inference of Semantic Heatmaps In Grids
- BEVCar: Camera-Radar Fusion for BEV Map and Object Segmentation
- FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras
- NeRF-Loc: Transformer-Based Object Localization Within Neural Radiance Fields
- U-BEV: Height-aware Bird's-Eye-View Segmentation and Neural Map-based Relocalization
- PrevPredMap: Exploring Temporal Modeling with Previous Predictions for Online Vectorized HD Map Construction
- DS-PASS: Detail-Sensitive Panoramic Annular Semantic Segmentation through SwaftNet for Surrounding Sensing
- DuMapNet: An End-to-End Vectorization System for City-Scale Lane-Level Map Generation
- LaRa: Latents and Rays for Multi-Camera Bird's-Eye-View Semantic Segmentation
- LDMapNet-U: An End-to-End System for City-Scale Lane-Level Map Updating
- Trust, but Verify: Cross-Modality Fusion for HD Map Change Detection
- RendBEV: Semantic Novel View Synthesis for Self-Supervised Bird's Eye View Segmentation
- HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor
- Learning a Decision Module by Imitating Driver's Control Behaviors
- DA-BEV: Unsupervised Domain Adaptation for Bird's Eye View Perception
- TLCFuse: Temporal Multi-Modality Fusion Towards Occlusion-Aware Semantic Segmentation-Aided Motion Planning
- Transfer beyond the Field of View: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation
- Capturing Omni-Range Context for Omnidirectional Segmentation
- Improved Single Camera BEV Perception Using Multi-Camera Training
- FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models