Causal Reasoning from Meta-reinforcement Learning

Abstract

Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents. Here we explore whether causal reasoning can emerge via meta-reinforcement learning. We train a recurrent network with model-free reinforcement learning to solve a range of problems that each contain causal structure. We find that the trained agent can perform causal reasoning in novel situations in order to obtain rewards. The agent can select informative interventions, draw causal inferences from observational data, and make counterfactual predictions. Although established formal causal reasoning algorithms also exist, in this paper we show that such reasoning can arise from model-free reinforcement learning, and suggest that causal reasoning in complex settings may benefit from the more end-to-end learning-based approaches presented here. This work also offers new strategies for structured exploration in reinforcement learning, by providing agents with the ability to perform -- and interpret -- experiments.

Cited by in corpus (24)

• A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
• Causality for Machine Learning
• Causal Induction from Visual Observations for Goal Directed Tasks
• Learning Neural Causal Models from Unknown Interventions
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Meta-learning of Sequential Strategies
• Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning
• A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning
• Algorithms for Causal Reasoning in Probability Trees
• Interventional Sum-Product Networks: Causal Inference with Tractable Probabilistic Models
• Towards Causality-Aware Inferring: A Sequential Discriminative Approach for Medical Diagnosis
• Meta-Learning of Structured Task Distributions in Humans and Machines
• Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
• Amortized learning of neural causal representations
• Disentangling causal effects for hierarchical reinforcement learning
• Group Equivariant Deep Reinforcement Learning
• Did I do that? Blame as a means to identify controlled effects in reinforcement learning
• Efficiently Disentangle Causal Representations
• Adversarial Examples and the Deeper Riddle of Induction: The Need for a Theory of Artifacts in Deep Learning
• Towards intervention-centric causal reasoning in learning agents
• Empirically Verifying Hypotheses Using Reinforcement Learning
• Where Do Human Heuristics Come From?