Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
arXiv:2106.03207
Abstract
This paper studies offline Imitation Learning (IL) where an agent learns to imitate an expert demonstrator without additional online environment interactions. Instead, the learner is presented with a static offline dataset of state-action-next state transition triples from a potentially less proficient behavior policy. We introduce Model-based IL from Offline data (MILO): an algorithmic framework that utilizes the static dataset to solve the offline IL problem efficiently both in theory and in practice. In theory, even if the behavior policy is highly sub-optimal compared to the expert, we show that as long as the data from the behavior policy provides sufficient coverage on the expert state-action traces (and with no necessity for a global coverage over the entire state-action space), MILO can provably combat the covariate shift issue in IL. Complementing our theory results, we also demonstrate that a practical implementation of our approach mitigates covariate shift on benchmark MuJoCo continuous control tasks. We demonstrate that with behavior policies whose performances are less than half of that of the expert, MILO still successfully imitates with an extremely low number of expert state-action pairs while traditional offline IL method such as behavior cloning (BC) fails completely. Source code is provided at https://github.com/jdchang1/milo.
42 pages, 5 figures, 7 tables
References in corpus (24)
- Conservative Q-Learning for Offline Reinforcement Learning
- Behavior Regularized Offline Reinforcement Learning
- Reinforcement and Imitation Learning via Interactive No-Regret Learning
- COMBO: Conservative Offline Model-Based Policy Optimization
- GenDICE: Generalized Offline Estimation of Stationary Values
- Information-Theoretic Considerations in Batch Reinforcement Learning
- Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
- What are the Statistical Limits of Offline RL with Linear Function Approximation?
- Imitation Learning via Off-Policy Distribution Matching
- PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning
- Information Theoretic Regret Bounds for Online Nonlinear Control
- Is Pessimism Provably Efficient for Offline RL?
- The Importance of Pessimism in Fixed-Dataset Policy Optimization
- Provably Efficient Imitation Learning from Observation Alone
- Toward the Fundamental Limits of Imitation Learning
- Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RL
- Near-Optimal Offline Reinforcement Learning via Double Variance Reduction
- Feedback in Imitation Learning: The Three Regimes of Covariate Shift
- Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning
- Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
- Learning to Search for Dependencies
- Scalable Bayesian Inverse Reinforcement Learning
- Continuous Doubly Constrained Batch Reinforcement Learning
- MobILE: Model-Based Imitation Learning From Observation Alone