3D-PhysNet: Learning the Intuitive Physics of Non-Rigid Object Deformations

Abstract

The ability to interact and understand the environment is a fundamental prerequisite for a wide range of applications from robotics to augmented reality. In particular, predicting how deformable objects will react to applied forces in real time is a significant challenge. This is further confounded by the fact that shape information about encountered objects in the real world is often impaired by occlusions, noise and missing regions e.g. a robot manipulating an object will only be able to observe a partial view of the entire solid. In this work we present a framework, 3D-PhysNet, which is able to predict how a three-dimensional solid will deform under an applied force using intuitive physics modelling. In particular, we propose a new method to encode the physical properties of the material and the applied force, enabling generalisation over materials. The key is to combine deep variational autoencoders with adversarial training, conditioned on the applied force and the material properties. We further propose a cascaded architecture that takes a single 2.5D depth view of the object and predicts its deformation. Training data is provided by a physics simulator. The network is fast enough to be used in real-time applications from partial views. Experimental results show the viability and the generalisation properties of the proposed architecture.

arXiv commentin IJCAI 2018

References in corpus (8)

• Improved Training of Wasserstein GANs
• Generating Videos with Scene Dynamics
• Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
• Unsupervised Learning for Physical Interaction through Video Prediction
• Improved Adversarial Systems for 3D Object Generation and Reconstruction
• CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training
• High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference
• 3D Object Reconstruction from a Single Depth View with Adversarial Learning

Cited by in corpus (7)

• Flexible Neural Representation for Physics Prediction
• When Deep Learning Meets Data Alignment: A Review on Deep Registration Networks (DRNs)
• Sparse Surface Constraints for Combining Physics-based Elasticity Simulation and Correspondence-Free Object Reconstruction
• Forward Prediction for Physical Reasoning
• Perceiving Physical Equation by Observing Visual Scenarios
• Neural Allocentric Intuitive Physics Prediction from Real Videos
• DispVoxNets: Non-Rigid Point Set Alignment with Supervised Learning Proxies