Occlusion resistant learning of intuitive physics from videos

Abstract

To reach human performance on complex tasks, a key ability for artificial systems is to understand physical interactions between objects, and predict future outcomes of a situation. This ability, often referred to as intuitive physics, has recently received attention and several methods were proposed to learn these physical rules from video sequences. Yet, most of these methods are restricted to the case where no, or only limited, occlusions occur. In this work we propose a probabilistic formulation of learning intuitive physics in 3D scenes with significant inter-object occlusions. In our formulation, object positions are modeled as latent variables enabling the reconstruction of the scene. We then propose a series of approximations that make this problem tractable. Object proposals are linked across frames using a combination of a recurrent interaction network, modeling the physics in object space, and a compositional renderer, modeling the way in which objects project onto pixel space. We demonstrate significant improvements over state-of-the-art in the intuitive physics benchmark of IntPhys. We apply our method to a second dataset with increasing levels of occlusions, showing it realistically predicts segmentation masks up to 30 frames in the future. Finally, we also show results on predicting motion of objects in real videos.

References in corpus (12)

• Interaction Networks for Learning about Objects, Relations and Physics
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Multi-Object Representation Learning with Iterative Variational Inference
• MONet: Unsupervised Scene Decomposition and Representation
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• Visual Interaction Networks
• To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction
• Learning A Physical Long-term Predictor
• A Comparative Evaluation of Approximate Probabilistic Simulation and Deep Neural Networks as Accounts of Human Physical Scene Understanding
• IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning
• Generalizable Features From Unsupervised Learning

Cited by in corpus (3)

• Physion: Evaluating Physical Prediction from Vision in Humans and Machines
• AlignNet: Unsupervised Entity Alignment
• AGENT: A Benchmark for Core Psychological Reasoning