CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations

Abstract

We propose CaSPR, a method to learn object-centric Canonical Spatiotemporal Point Cloud Representations of dynamically moving or evolving objects. Our goal is to enable information aggregation over time and the interrogation of object state at any spatiotemporal neighborhood in the past, observed or not. Different from previous work, CaSPR learns representations that support spacetime continuity, are robust to variable and irregularly spacetime-sampled point clouds, and generalize to unseen object instances. Our approach divides the problem into two subtasks. First, we explicitly encode time by mapping an input point cloud sequence to a spatiotemporally-canonicalized object space. We then leverage this canonicalization to learn a spatiotemporal latent representation using neural ordinary differential equations and a generative model of dynamically evolving shapes using continuous normalizing flows. We demonstrate the effectiveness of our method on several applications including shape reconstruction, camera pose estimation, continuous spatiotemporal sequence reconstruction, and correspondence estimation from irregularly or intermittently sampled observations.

NeurIPS 2020

References in corpus (17)

• Variational Inference with Normalizing Flows
• RPM-Net: Robust Point Matching using Learned Features
• Learning Representations and Generative Models for 3D Point Clouds
• Normalizing Flows for Probabilistic Modeling and Inference
• Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models
• High-Dimensional Probability Estimation with Deep Density Models
• Normalizing Flows on Riemannian Manifolds
• Equivariant Flows: sampling configurations for multi-body systems with symmetric energies
• Equivariant Hamiltonian Flows
• Graph Normalizing Flows
• Monge-Ampère Flow for Generative Modeling
• Continuous-Time Flows for Efficient Inference and Density Estimation
• Multiview Aggregation for Learning Category-Specific Shape Reconstruction
• Neural Manifold Ordinary Differential Equations
• Semi-Conditional Normalizing Flows for Semi-Supervised Learning
• Deep Diffeomorphic Normalizing Flows
• Time Dependence in Non-Autonomous Neural ODEs

Cited by in corpus (6)

• Physics-informed PointNet: A deep learning solver for steady-state incompressible flows and thermal fields on multiple sets of irregular geometries
• Point-Cloud Deep Learning of Porous Media for Permeability Prediction
• Learning Neural Event Functions for Ordinary Differential Equations
• Neural Spatio-Temporal Point Processes
• Neural Fields in Visual Computing and Beyond
• VidStyleODE: Disentangled Video Editing via StyleGAN and NeuralODEs