Co-Fusion: Real-time Segmentation, Tracking and Fusion of Multiple Objects
arXiv:1706.06629 · doi:10.1109/ICRA.2017.7989518
Abstract
In this paper we introduce Co-Fusion, a dense SLAM system that takes a live stream of RGB-D images as input and segments the scene into different objects (using either motion or semantic cues) while simultaneously tracking and reconstructing their 3D shape in real time. We use a multiple model fitting approach where each object can move independently from the background and still be effectively tracked and its shape fused over time using only the information from pixels associated with that object label. Previous attempts to deal with dynamic scenes have typically considered moving regions as outliers, and consequently do not model their shape or track their motion over time. In contrast, we enable the robot to maintain 3D models for each of the segmented objects and to improve them over time through fusion. As a result, our system can enable a robot to maintain a scene description at the object level which has the potential to allow interactions with its working environment; even in the case of dynamic scenes.
arXiv commentInternational Conference on Robotics and Automation (ICRA) 2017, http://visual.cs.ucl.ac.uk/pubs/cofusion, https://github.com/martinruenz/co-fusion
References in corpus (2)
Cited by in corpus (39)
- Robust Dense Mapping for Large-Scale Dynamic Environments
- Semantics for Robotic Mapping, Perception and Interaction: A Survey
- A Robust Laser-Inertial Odometry and Mapping Method for Large-Scale Highway Environments
- Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution Volumetric Mapping and Long-term Dynamic Scene Consistency
- EM-Fusion: Dynamic Object-Level SLAM with Probabilistic Data Association
- Multimotion Visual Odometry (MVO): Simultaneous Estimation of Camera and Third-Party Motions
- TwistSLAM: Constrained SLAM in Dynamic Environment
- A Survey of Simultaneous Localization and Mapping with an Envision in 6G Wireless Networks
- FutureMapping: The Computational Structure of Spatial AI Systems
- Topology-Aware Non-Rigid Point Cloud Registration
- RGB-D SLAM in Indoor Planar Environments with Multiple Large Dynamic Objects
- Context-Aware Mixed Reality: A Framework for Ubiquitous Interaction
- MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM
- FlowFusion: Dynamic Dense RGB-D SLAM Based on Optical Flow
- RGB-D SLAM in Dynamic Environments Using Point Correlations
- Kimera-Multi: a System for Distributed Multi-Robot Metric-Semantic Simultaneous Localization and Mapping
- Multimotion Visual Odometry (MVO)
- DenseSplat: Densifying Gaussian Splatting SLAM with Neural Radiance Prior
- Using Detection, Tracking and Prediction in Visual SLAM to Achieve Real-time Semantic Mapping of Dynamic Scenarios
- SurfelWarp: Efficient Non-Volumetric Single View Dynamic Reconstruction
- Efficient 3D Reconstruction, Streaming and Visualization of Static and Dynamic Scene Parts for Multi-client Live-telepresence in Large-scale Environments
- Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems
- ClusterVO: Clustering Moving Instances and Estimating Visual Odometry for Self and Surroundings
- Temporally Consistent Object 6D Pose Estimation for Robot Control
- Where Does It End? -- Reasoning About Hidden Surfaces by Object Intersection Constraints
- Building 3D Object Models during Manipulation by Reconstruction-Aware Trajectory Optimization
- DetectFusion: Detecting and Segmenting Both Known and Unknown Dynamic Objects in Real-time SLAM
- RigidFusion: Robot Localisation and Mapping in Environments with Large Dynamic Rigid Objects
- DymSLAM:4D Dynamic Scene Reconstruction Based on Geometrical Motion Segmentation
- Physics-Based Rigid Body Object Tracking and Friction Filtering From RGB-D Videos
- Extracting Contact and Motion from Manipulation Videos
- SplitFusion: Simultaneous Tracking and Mapping for Non-Rigid Scenes
- TivNe-SLAM: Dynamic Mapping and Tracking via Time-Varying Neural Radiance Fields
- DAL -- A Deep Depth-aware Long-term Tracker
- Live Reconstruction of Large-Scale Dynamic Outdoor Worlds
- MOLTR: Multiple Object Localisation, Tracking, and Reconstruction from Monocular RGB Videos
- Sparse-Dense Motion Modelling and Tracking for Manipulation without Prior Object Models
- Vision-based deep execution monitoring
- 3D Siamese Voxel-to-BEV Tracker for Sparse Point Clouds