3D Shape Reconstruction from Vision and Touch
arXiv:2007.03778
Abstract
When a toddler is presented a new toy, their instinctual behaviour is to pick it upand inspect it with their hand and eyes in tandem, clearly searching over its surface to properly understand what they are playing with. At any instance here, touch provides high fidelity localized information while vision provides complementary global context. However, in 3D shape reconstruction, the complementary fusion of visual and haptic modalities remains largely unexplored. In this paper, we study this problem and present an effective chart-based approach to multi-modal shape understanding which encourages a similar fusion vision and touch information.To do so, we introduce a dataset of simulated touch and vision signals from the interaction between a robotic hand and a large array of 3D objects. Our results show that (1) leveraging both vision and touch signals consistently improves single-modality baselines; (2) our approach outperforms alternative modality fusion methods and strongly benefits from the proposed chart-based structure; (3) there construction quality increases with the number of grasps provided; and (4) the touch information not only enhances the reconstruction at the touch site but also extrapolates to its local neighborhood.
arXiv commentAccepted at Neurips 2020
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
- Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
- MarrNet: 3D Shape Reconstruction via 2.5D Sketches
- Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images
- Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research
- Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
- GEOMetrics: Exploiting Geometric Structure for Graph-Encoded Objects
- Learning Category-Specific Mesh Reconstruction from Image Collections
- Hierarchical Surface Prediction for 3D Object Reconstruction
- Learning to Generate and Reconstruct 3D Meshes with only 2D Supervision
- Unsupervised Learning of Shape and Pose with Differentiable Point Clouds
- Robotic Tactile Perception of Object Properties: A Review
- Neural Multisensory Scene Inference