Learning 3D Dynamic Scene Representations for Robot Manipulation
arXiv:2011.01968
Abstract
3D scene representation for robot manipulation should capture three key object properties: permanency -- objects that become occluded over time continue to exist; amodal completeness -- objects have 3D occupancy, even if only partial observations are available; spatiotemporal continuity -- the movement of each object is continuous over space and time. In this paper, we introduce 3D Dynamic Scene Representation (DSR), a 3D volumetric scene representation that simultaneously discovers, tracks, reconstructs objects, and predicts their dynamics while capturing all three properties. We further propose DSR-Net, which learns to aggregate visual observations over multiple interactions to gradually build and refine DSR. Our model achieves state-of-the-art performance in modeling 3D scene dynamics with DSR on both simulated and real data. Combined with model predictive control, DSR-Net enables accurate planning in downstream robotic manipulation tasks such as planar pushing. Video is available at https://youtu.be/GQjYG3nQJ80.
CoRL 2020. The first two authors contributed equally to this paper. Project page: https://dsr-net.cs.columbia.edu/
References in corpus (11)
- SfM-Net: Learning of Structure and Motion from Video
- Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
- Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
- MONet: Unsupervised Scene Decomposition and Representation
- Learning to Infer Implicit Surfaces without 3D Supervision
- Spatial Action Maps for Mobile Manipulation
- SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Planning and Control
- Geometry-Aware Recurrent Neural Networks for Active Visual Recognition
- Learning from Unlabelled Videos Using Contrastive Predictive Neural 3D Mapping
- Object-centric Forward Modeling for Model Predictive Control
- Learning Visual Dynamics Models of Rigid Objects using Relational Inductive Biases