Evaluating task-agnostic exploration for fixed-batch learning of arbitrary future tasks

Abstract

Deep reinforcement learning has been shown to solve challenging tasks where large amounts of training experience is available, usually obtained online while learning the task. Robotics is a significant potential application domain for many of these algorithms, but generating robot experience in the real world is expensive, especially when each task requires a lengthy online training procedure. Off-policy algorithms can in principle learn arbitrary tasks from a diverse enough fixed dataset. In this work, we evaluate popular exploration methods by generating robotics datasets for the purpose of learning to solve tasks completely offline without any further interaction in the real world. We present results on three popular continuous control tasks in simulation, as well as continuous control of a high-dimensional real robot arm. Code documenting all algorithms, experiments, and hyper-parameters is available at https://github.com/qutrobotlearning/batchlearning.

References in corpus (10)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Large-Scale Study of Curiosity-Driven Learning
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Model-Ensemble Trust-Region Policy Optimization
• Exploration by Random Network Distillation
• Towards Vision-Based Deep Reinforcement Learning for Robotic Motion Control
• Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
• One-Shot Reinforcement Learning for Robot Navigation with Interactive Replay