Hybrid Reinforcement Learning with Expert State Sequences
arXiv:1903.04110
Abstract
Existing imitation learning approaches often require that the complete demonstration data, including sequences of actions and states, are available. In this paper, we consider a more realistic and difficult scenario where a reinforcement learning agent only has access to the state sequences of an expert, while the expert actions are unobserved. We propose a novel tensor-based model to infer the unobserved actions of the expert state sequences. The policy of the agent is then optimized via a hybrid objective combining reinforcement learning and imitation learning. We evaluated our hybrid approach on an illustrative domain and Atari games. The empirical results show that (1) the agents are able to leverage state expert sequences to learn faster than pure reinforcement learning baselines, (2) our tensor-based action inference model is advantageous compared to standard deep neural networks in inferring expert actions, and (3) the hybrid policy optimization objective is robust against noise in expert state sequences.
arXiv commentAAAI 2019; https://github.com/XiaoxiaoGuo/tensor4rl
References in corpus (10)
- Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
- Action-Conditional Video Prediction using Deep Networks in Atari Games
- Generative Adversarial Imitation Learning
- One-Shot Imitation Learning
- Third-Person Imitation Learning
- One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
- Reinforcement and Imitation Learning for Diverse Visuomotor Skills
- Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay
- Visual Question Generation as Dual Task of Visual Question Answering
- Observational Learning by Reinforcement Learning