Causal Discovery in Physical Systems from Videos

Abstract

Causal discovery is at the core of human cognition. It enables us to reason about the environment and make counterfactual predictions about unseen scenarios that can vastly differ from our previous experiences. We consider the task of causal discovery from videos in an end-to-end fashion without supervision on the ground-truth graph structure. In particular, our goal is to discover the structural dependencies among environmental and object variables: inferring the type and strength of interactions that have a causal effect on the behavior of the dynamical system. Our model consists of (a) a perception module that extracts a semantically meaningful and temporally consistent keypoint representation from images, (b) an inference module for determining the graph distribution induced by the detected keypoints, and (c) a dynamics module that can predict the future by conditioning on the inferred graph. We assume access to different configurations and environmental conditions, i.e., data from unknown interventions on the underlying system; thus, we can hope to discover the correct underlying causal graph without explicit interventions. We evaluate our method in a planar multi-body interaction environment and scenarios involving fabrics of different shapes like shirts and pants. Experiments demonstrate that our model can correctly identify the interactions from a short sequence of images and make long-term future predictions. The causal structure assumed by the model also allows it to make counterfactual predictions and extrapolate to systems of unseen interaction graphs or graphs of various sizes.

arXiv commentNeurIPS 2020. Project page: https://yunzhuli.github.io/V-CDN/

References in corpus (24)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Auto-Encoding Variational Bayes
• Categorical Reparameterization with Gumbel-Softmax
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Interaction Networks for Learning about Objects, Relations and Physics
• Learning Latent Dynamics for Planning from Pixels
• Graph networks as learnable physics engines for inference and control
• On the Identifiability of the Post-Nonlinear Causal Model
• DAGs with NO TEARS: Continuous Optimization for Structure Learning
• Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
• Flexible Neural Representation for Physics Prediction
• Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning
• Unsupervised Learning of Object Keypoints for Perception and Control
• Unsupervised Learning of Object Landmarks through Conditional Image Generation
• Recurrent Independent Mechanisms
• Permutation-based Causal Inference Algorithms with Interventions
• Learning Compositional Koopman Operators for Model-Based Control
• Learning Neural Causal Models from Unknown Interventions
• Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning
• backShift: Learning causal cyclic graphs from unknown shift interventions
• Structural Agnostic Modeling: Adversarial Learning of Causal Graphs
• Neural Relational Inference with Fast Modular Meta-learning
• Causal Generative Neural Networks
• Causal Discovery in the Presence of Measurement Error: Identifiability Conditions

Cited by in corpus (9)

• Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data
• 3D Neural Scene Representations for Visuomotor Control
• Multi-Task Reinforcement Learning with Context-based Representations
• ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind
• Learning Temporally Causal Latent Processes from General Temporal Data
• Learning latent causal graphs via mixture oracles
• Beyond permutation equivariance in graph networks
• Self-supervised Reinforcement Learning with Independently Controllable Subgoals
• Causal Discovery from Conditionally Stationary Time Series