Model Imitation for Model-Based Reinforcement Learning

Abstract

Model-based reinforcement learning (MBRL) aims to learn a dynamic model to reduce the number of interactions with real-world environments. However, due to estimation error, rollouts in the learned model, especially those of long horizons, fail to match the ones in real-world environments. This mismatching has seriously impacted the sample complexity of MBRL. The phenomenon can be attributed to the fact that previous works employ supervised learning to learn the one-step transition models, which has inherent difficulty ensuring the matching of distributions from multi-step rollouts. Based on the claim, we propose to learn the transition model by matching the distributions of multi-step rollouts sampled from the transition model and the real ones via WGAN. We theoretically show that matching the two can minimize the difference of cumulative rewards between the real transition and the learned one. Our experiments also show that the proposed Model Imitation method can compete or outperform the state-of-the-art in terms of sample complexity and average return.

References in corpus (18)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Spectral Normalization for Generative Adversarial Networks
• Trust Region Policy Optimization
• Self-Attention Generative Adversarial Networks
• Addressing Function Approximation Error in Actor-Critic Methods
• Frustratingly Easy Domain Adaptation
• Gaussian Processes for Data-Efficient Learning in Robotics and Control
• Model-Ensemble Trust-Region Policy Optimization
• Benchmarking Model-Based Reinforcement Learning
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Geometric GAN
• Learning Probability Measures with respect to Optimal Transport Metrics
• Combating the Compounding-Error Problem with a Multi-step Model
• Self-Correcting Models for Model-Based Reinforcement Learning
• MineGAN: effective knowledge transfer from GANs to target domains with few images

Cited by in corpus (3)

• Model-based Policy Optimization with Unsupervised Model Adaptation
• Active World Model Learning with Progress Curiosity
• Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning