Automating Reinforcement Learning with Example-based Resets

Abstract

Deep reinforcement learning has enabled robots to learn motor skills from environmental interactions with minimal to no prior knowledge. However, existing reinforcement learning algorithms assume an episodic setting, in which the agent resets to a fixed initial state distribution at the end of each episode, to successfully train the agents from repeated trials. Such reset mechanism, while trivial for simulated tasks, can be challenging to provide for real-world robotics tasks. Resets in robotic systems often require extensive human supervision and task-specific workarounds, which contradicts the goal of autonomous robot learning. In this paper, we propose an extension to conventional reinforcement learning towards greater autonomy by introducing an additional agent that learns to reset in a self-supervised manner. The reset agent preemptively triggers a reset to prevent manual resets and implicitly imposes a curriculum for the forward agent. We apply our method to learn from scratch on a suite of simulated and real-world continuous control tasks and demonstrate that the reset agent successfully learns to reduce manual resets whilst also allowing the forward policy to improve gradually over time.

arXiv comment8 pages, 6 figures; accepted for publication in the IEEE Robotics and Automation Letters (RA-L); source code available at https://github.com/jigangkim/autoreset_rl ; supplementary video available at https://youtu.be/himd0Z5b64A

References in corpus (10)

• Prioritized Experience Replay
• Dota 2 with Large Scale Deep Reinforcement Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• DeepMind Control Suite
• Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search
• Randomized Prior Functions for Deep Reinforcement Learning
• Learning to Walk in the Real World with Minimal Human Effort
• Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement

Cited by in corpus (1)

• Real-Time Reinforcement Learning for Dynamic Tasks with a Parallel Soft Robot