E3D: Event-Based 3D Shape Reconstruction
arXiv:2012.05214
Abstract
3D shape reconstruction is a primary component of augmented/virtual reality. Despite being highly advanced, existing solutions based on RGB, RGB-D and Lidar sensors are power and data intensive, which introduces challenges for deployment in edge devices. We approach 3D reconstruction with an event camera, a sensor with significantly lower power, latency and data expense while enabling high dynamic range. While previous event-based 3D reconstruction methods are primarily based on stereo vision, we cast the problem as multi-view shape from silhouette using a monocular event camera. The output from a moving event camera is a sparse point set of space-time gradients, largely sketching scene/object edges and contours. We first introduce an event-to-silhouette (E2S) neural network module to transform a stack of event frames to the corresponding silhouettes, with additional neural branches for camera pose regression. Second, we introduce E3D, which employs a 3D differentiable renderer (PyTorch3D) to enforce cross-view 3D mesh consistency and fine-tune the E2S and pose network. Lastly, we introduce a 3D-to-events simulation pipeline and apply it to publicly available object datasets and generate synthetic event/silhouette training pairs for supervised learning.
Correct author names and only include primary author email
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- Accelerating 3D Deep Learning with PyTorch3D
- InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset
- Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction
- Unsupervised Learning of Dense Optical Flow, Depth and Egomotion from Sparse Event Data
- Collaborative Video Object Segmentation by Foreground-Background Integration