RPM-Net: Recurrent Prediction of Motion and Parts from Point Cloud

Abstract

We introduce RPM-Net, a deep learning-based approach which simultaneously infers movable parts and hallucinates their motions from a single, un-segmented, and possibly partial, 3D point cloud shape. RPM-Net is a novel Recurrent Neural Network (RNN), composed of an encoder-decoder pair with interleaved Long Short-Term Memory (LSTM) components, which together predict a temporal sequence of pointwise displacements for the input point cloud. At the same time, the displacements allow the network to learn movable parts, resulting in a motion-based shape segmentation. Recursive applications of RPM-Net on the obtained parts can predict finer-level part motions, resulting in a hierarchical object segmentation. Furthermore, we develop a separate network to estimate part mobilities, e.g., per-part motion parameters, from the segmented motion sequence. Both networks learn deep predictive models from a training set that exemplifies a variety of mobilities for diverse objects. We show results of simultaneous motion and part predictions from synthetic and real scans of 3D objects exhibiting a variety of part mobilities, possibly involving multiple movable parts.

Accepted to SIGGRAPH Asia 2019, project page at https://vcc.tech/research/2019/RPMNet

Cited by in corpus (7)

• Assemble Them All: Physics-Based Planning for Generalizable Assembly by Disassembly
• ClipGen: A Deep Generative Model for Clipart Vectorization and Synthesis
• D3D-HOI: Dynamic 3D Human-Object Interactions from Videos
• Morig: Motion-aware rigging of character meshes from point clouds
• Shape Completion with Points in the Shadow
• MultiBodySync: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronization
• Distributional Depth-Based Estimation of Object Articulation Models