Minimax Model Learning

Abstract

We present a novel off-policy loss function for learning a transition model in model-based reinforcement learning. Notably, our loss is derived from the off-policy policy evaluation objective with an emphasis on correcting distribution shift. Compared to previous model-based techniques, our approach allows for greater robustness under model misspecification or distribution shift induced by learning/evaluating policies that are distinct from the data-generating policy. We provide a theoretical analysis and show empirical improvements over existing model-based off-policy evaluation methods. We provide further analysis showing our loss can be used for off-policy optimization (OPO) and demonstrate its integration with more recent improvements in OPO.

References in corpus (9)

• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• When to Trust Your Model: Model-Based Policy Optimization
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• A Game Theoretic Framework for Model Based Reinforcement Learning
• Information-Theoretic Considerations in Batch Reinforcement Learning
• A Kernel Loss for Solving the Bellman Equation
• Policy-Aware Model Learning for Policy Gradient Methods