Reasoning About Physical Interactions with Object-Oriented Prediction and Planning

Abstract

Object-based factorizations provide a useful level of abstraction for interacting with the world. Building explicit object representations, however, often requires supervisory signals that are difficult to obtain in practice. We present a paradigm for learning object-centric representations for physical scene understanding without direct supervision of object properties. Our model, Object-Oriented Prediction and Planning (O2P2), jointly learns a perception function to map from image observations to object representations, a pairwise physics interaction function to predict the time evolution of a collection of objects, and a rendering function to map objects back to pixels. For evaluation, we consider not only the accuracy of the physical predictions of the model, but also its utility for downstream tasks that require an actionable representation of intuitive physics. After training our model on an image prediction task, we can use its learned representations to build block towers more complicated than those observed during training.

ICLR 2019, project page: https://people.eecs.berkeley.edu/~janner/o2p2/

References in corpus (3)

• Stochastic Adversarial Video Prediction
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Learning Physical Intuition of Block Towers by Example

Cited by in corpus (24)

• The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Learning Compositional Koopman Operators for Model-Based Control
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
• Learn to Predict How Humans Manipulate Large-sized Objects from Interactive Motions
• Visual Grounding of Learned Physical Models
• O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning
• Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video
• Physion: Evaluating Physical Prediction from Vision in Humans and Machines
• Learning Physical Constraints with Neural Projections
• Unsupervised Discovery of 3D Physical Objects from Video
• AlignNet: Unsupervised Entity Alignment
• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
• Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
• Forward Prediction for Physical Reasoning
• Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
• AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions
• PHYRE: A New Benchmark for Physical Reasoning
• Stable Object Reorientation using Contact Plane Registration
• Evaluating the Apperception Engine
• Pushing it out of the Way: Interactive Visual Navigation