Instabilities of Offline RL with Pre-Trained Neural Representation

Abstract

In offline reinforcement learning (RL), we seek to utilize offline data to evaluate (or learn) policies in scenarios where the data are collected from a distribution that substantially differs from that of the target policy to be evaluated. Recent theoretical advances have shown that such sample-efficient offline RL is indeed possible provided certain strong representational conditions hold, else there are lower bounds exhibiting exponential error amplification (in the problem horizon) unless the data collection distribution has only a mild distribution shift relative to the target policy. This work studies these issues from an empirical perspective to gauge how stable offline RL methods are. In particular, our methodology explores these ideas when using features from pre-trained neural networks, in the hope that these representations are powerful enough to permit sample efficient offline RL. Through extensive experiments on a range of tasks, we see that substantial error amplification does occur even when using such pre-trained representations (trained on the same task itself); we find offline RL is stable only under extremely mild distribution shift. The implications of these results, both from a theoretical and an empirical perspective, are that successful offline RL (where we seek to go beyond the low distribution shift regime) requires substantially stronger conditions beyond those which suffice for successful supervised learning.

References in corpus (24)

• Continuous control with deep reinforcement learning
• Addressing Function Approximation Error in Actor-Critic Methods
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• VIME: Variational Information Maximizing Exploration
• Doubly Robust Policy Evaluation and Learning
• MOPO: Model-based Offline Policy Optimization
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Evaluating Reinforcement Learning Algorithms in Observational Health Settings
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• COMBO: Conservative Offline Model-Based Policy Optimization
• Information-Theoretic Considerations in Batch Reinforcement Learning
• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• Batch Value-function Approximation with Only Realizability
• Minimax Weight and Q-Function Learning for Off-Policy Evaluation
• Semantic Visual Navigation by Watching YouTube Videos
• Off-Policy Evaluation via the Regularized Lagrangian
• Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RL
• Offline Reinforcement Learning from Images with Latent Space Models
• Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
• A Variant of the Wang-Foster-Kakade Lower Bound for the Discounted Setting
• Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Sampling
• Accountable Off-Policy Evaluation With Kernel Bellman Statistics

Cited by in corpus (4)

• Offline RL Without Off-Policy Evaluation
• Bellman-consistent Pessimism for Offline Reinforcement Learning
• Offline Reinforcement Learning with Value-based Episodic Memory
• The Difficulty of Passive Learning in Deep Reinforcement Learning