Maximum Entropy Model-based Reinforcement Learning

Abstract

Recent advances in reinforcement learning have demonstrated its ability to solve hard agent-environment interaction tasks on a super-human level. However, the application of reinforcement learning methods to practical and real-world tasks is currently limited due to most RL state-of-art algorithms' sample inefficiency, i.e., the need for a vast number of training episodes. For example, OpenAI Five algorithm that has beaten human players in Dota 2 has trained for thousands of years of game time. Several approaches exist that tackle the issue of sample inefficiency, that either offers a more efficient usage of already gathered experience or aim to gain a more relevant and diverse experience via a better exploration of an environment. However, to our knowledge, no such approach exists for model-based algorithms, that showed their high sample efficiency in solving hard control tasks with high-dimensional state space. This work connects exploration techniques and model-based reinforcement learning. We have designed a novel exploration method that takes into account features of the model-based approach. We also demonstrate through experiments that our method significantly improves the performance of the model-based algorithm Dreamer.

arXiv commentNeurIPS'2021 Deep Reinforcement Learning Workshop

References in corpus (8)

• Continuous control with deep reinforcement learning
• Dota 2 with Large Scale Deep Reinforcement Learning
• Solving Rubik's Cube with a Robot Hand
• Learning Latent Dynamics for Planning from Pixels
• Exploration by Random Network Distillation
• Provably Efficient Maximum Entropy Exploration
• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• Latent World Models For Intrinsically Motivated Exploration