Offline Reinforcement Learning with Reverse Model-based Imagination

Abstract

In offline reinforcement learning (offline RL), one of the main challenges is to deal with the distributional shift between the learning policy and the given dataset. To address this problem, recent offline RL methods attempt to introduce conservatism bias to encourage learning in high-confidence areas. Model-free approaches directly encode such bias into policy or value function learning using conservative regularizations or special network structures, but their constrained policy search limits the generalization beyond the offline dataset. Model-based approaches learn forward dynamics models with conservatism quantifications and then generate imaginary trajectories to extend the offline datasets. However, due to limited samples in offline datasets, conservatism quantifications often suffer from overgeneralization in out-of-support regions. The unreliable conservative measures will mislead forward model-based imaginations to undesired areas, leading to overaggressive behaviors. To encourage more conservatism, we propose a novel model-based offline RL framework, called Reverse Offline Model-based Imagination (ROMI). We learn a reverse dynamics model in conjunction with a novel reverse policy, which can generate rollouts leading to the target goal states within the offline dataset. These reverse imaginations provide informed data augmentation for model-free policy learning and enable conservative generalization beyond the offline dataset. ROMI can effectively combine with off-the-shelf model-free algorithms to enable model-based generalization with proper conservatism. Empirical results show that our method can generate more conservative behaviors and achieve state-of-the-art performance on offline RL benchmark tasks.

References in corpus (12)

• Conservative Q-Learning for Offline Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• COMBO: Conservative Offline Model-Based Policy Optimization
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• Provably Good Batch Reinforcement Learning Without Great Exploration
• Offline Reinforcement Learning from Images with Latent Space Models
• PLAS: Latent Action Space for Offline Reinforcement Learning
• Bidirectional Model-based Policy Optimization