Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
arXiv:2104.13877
Abstract
Standard dynamics models for continuous control make use of feedforward computation to predict the conditional distribution of next state and reward given current state and action using a multivariate Gaussian with a diagonal covariance structure. This modeling choice assumes that different dimensions of the next state and reward are conditionally independent given the current state and action and may be driven by the fact that fully observable physics-based simulation environments entail deterministic transition dynamics. In this paper, we challenge this conditional independence assumption and propose a family of expressive autoregressive dynamics models that generate different dimensions of the next state and reward sequentially conditioned on previous dimensions. We demonstrate that autoregressive dynamics models indeed outperform standard feedforward models in log-likelihood on heldout transitions. Furthermore, we compare different model-based and model-free off-policy evaluation (OPE) methods on RL Unplugged, a suite of offline MuJoCo datasets, and find that autoregressive dynamics models consistently outperform all baselines, achieving a new state-of-the-art. Finally, we show that autoregressive dynamics models are useful for offline policy optimization by serving as a way to enrich the replay buffer through data augmentation and improving performance using model-based planning.
arXiv commentICLR 2021. 17 pages
References in corpus (21)
- Adam: A Method for Stochastic Optimization
- Language Models are Few-Shot Learners
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
- Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
- Distributed Distributional Deterministic Policy Gradients
- Doubly Robust Policy Evaluation and Learning
- Model-Ensemble Trust-Region Policy Optimization
- Behavior Regularized Offline Reinforcement Learning
- MOPO: Model-based Offline Policy Optimization
- Benchmarking Model-Based Reinforcement Learning
- DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
- Critic Regularized Regression
- Model Predictive Path Integral Control using Covariance Variable Importance Sampling
- Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
- Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
- Minimax Weight and Q-Function Learning for Off-Policy Evaluation
- Off-Policy Evaluation via the Regularized Lagrangian
- Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation
- On Minimax Optimal Offline Policy Evaluation
- Batch Stationary Distribution Estimation