Causal Reinforcement Learning using Observational and Interventional Data

Abstract

Learning efficiently a causal model of the environment is a key challenge of model-based RL agents operating in POMDPs. We consider here a scenario where the learning agent has the ability to collect online experiences through direct interactions with the environment (interventional data), but has also access to a large collection of offline experiences, obtained by observing another agent interacting with the environment (observational data). A key ingredient, that makes this situation non-trivial, is that we allow the observed agent to interact with the environment based on hidden information, which is not observed by the learning agent. We then ask the following questions: can the online and offline experiences be safely combined for learning a causal model ? And can we expect the offline experiences to improve the agent's performances ? To answer these questions, we import ideas from the well-established causal framework of do-calculus, and we express model-based reinforcement learning as a causal inference problem. Then, we propose a general yet simple methodology for leveraging offline data during learning. In a nutshell, the method relies on learning a latent-based causal transition model that explains both the interventional and observational regimes, and then using the recovered latent variable to infer the standard POMDP transition model via deconfounding. We prove our method is correct and efficient in the sense that it attains better generalization guarantees due to the offline data (in the asymptotic case), and we illustrate its effectiveness empirically on synthetic toy problems. Our contribution aims at bridging the gap between the fields of reinforcement learning and causality.

References in corpus (6)

• Zero-Shot Text-to-Image Generation
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Pearl's Calculus of Intervention Is Complete
• Deconfounding Reinforcement Learning in Observational Settings
• Feedback in Imitation Learning: The Three Regimes of Covariate Shift
• Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders

Cited by in corpus (4)

• Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena
• A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Causal Multi-Agent Reinforcement Learning: Review and Open Problems