Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids

Abstract

Real-life control tasks involve matters of various substances---rigid or soft bodies, liquid, gas---each with distinct physical behaviors. This poses challenges to traditional rigid-body physics engines. Particle-based simulators have been developed to model the dynamics of these complex scenes; however, relying on approximation techniques, their simulation often deviates from real-world physics, especially in the long term. In this paper, we propose to learn a particle-based simulator for complex control tasks. Combining learning with particle-based systems brings in two major benefits: first, the learned simulator, just like other particle-based systems, acts widely on objects of different materials; second, the particle-based representation poses strong inductive bias for learning: particles of the same type have the same dynamics within. This enables the model to quickly adapt to new environments of unknown dynamics within a few observations. We demonstrate robots achieving complex manipulation tasks using the learned simulator, such as manipulating fluids and deformable foam, with experiments both in simulation and in the real world. Our study helps lay the foundation for robot learning of dynamic scenes with particle-based representations.

arXiv commentAccepted to ICLR 2019. Project Page: http://dpi.csail.mit.edu Video: https://www.youtube.com/watch?v=FrPpP7aW3Lg

References in corpus (10)

• Continuous Deep Q-Learning with Model-based Acceleration
• Graph networks as learnable physics engines for inference and control
• Imagination-Augmented Agents for Deep Reinforcement Learning
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Flexible Neural Representation for Physics Prediction
• The Predictron: End-To-End Learning and Planning
• Universal Planning Networks
• Learning model-based planning from scratch
• Visual Interaction Networks
• A Differentiable Physics Engine for Deep Learning in Robotics

Cited by in corpus (38)

• Learning to Simulate Complex Physics with Graph Networks
• Discovering Symbolic Models from Deep Learning with Inductive Biases
• Multipole Graph Neural Operator for Parametric Partial Differential Equations
• Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers
• What Can Neural Networks Reason About?
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contact
• DiffAqua: A Differentiable Computational Design Pipeline for Soft Underwater Swimmers with Shape Interpolation
• Teaching the Incompressible Navier-Stokes Equations to Fast Neural Surrogate Models in 3D
• Learning Compositional Koopman Operators for Model-Based Control
• Learning Mesh-Based Simulation with Graph Networks
• Causal Discovery in Physical Systems from Videos
• FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding
• Fluid Simulation on Neural Flow Maps
• Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
• Interactive Differentiable Simulation
• Predicting fluid-structure interaction with graph neural networks
• Visual-Tactile Multimodality for Following Deformable Linear Objects Using Reinforcement Learning
• Manipulation of Granular Materials by Learning Particle Interactions
• Visual Grounding of Learned Physical Models
• Graph Element Networks: adaptive, structured computation and memory
• Learning Visible Connectivity Dynamics for Cloth Smoothing
• Towards Complex Dynamic Physics System Simulation with Graph Neural ODEs
• Physion: Evaluating Physical Prediction from Vision in Humans and Machines
• Predicting Rigid Body Dynamics using Dual Quaternion Recurrent Neural Networks with Quaternion Attention
• Learning Physical Constraints with Neural Projections
• MPMNet: A Data-Driven MPM Framework for Dynamic Fluid-Solid Interaction
• MORPH-DSLAM: Model Order Reduction for PHysics-based Deformable SLAM
• Forward Prediction for Physical Reasoning
• Contrastive Variational Reinforcement Learning for Complex Observations
• Learning Incompressible Fluid Dynamics from Scratch -- Towards Fast, Differentiable Fluid Models that Generalize
• SENSORIMOTOR GRAPH: Action-Conditioned Graph Neural Network for Learning Robotic Soft Hand Dynamics
• Tranquil Clouds: Neural Networks for Learning Temporally Coherent Features in Point Clouds
• Follow the Object: Curriculum Learning for Manipulation Tasks with Imagined Goals
• Dynamic Relational Inference in Multi-Agent Trajectories
• Irregular Convolutional Auto-Encoder on Point Clouds
• Continuous Perception for Classifying Shapes and Weights of Garmentsfor Robotic Vision Applications
• Subspace Graph Physics: Real-Time Rigid Body-Driven Granular Flow Simulation