Learning by Playing - Solving Sparse Reward Tasks from Scratch

Abstract

We propose Scheduled Auxiliary Control (SAC-X), a new learning paradigm in the context of Reinforcement Learning (RL). SAC-X enables learning of complex behaviors - from scratch - in the presence of multiple sparse reward signals. To this end, the agent is equipped with a set of general auxiliary tasks, that it attempts to learn simultaneously via off-policy RL. The key idea behind our method is that active (learned) scheduling and execution of auxiliary policies allows the agent to efficiently explore its environment - enabling it to excel at sparse reward RL. Our experiments in several challenging robotic manipulation settings demonstrate the power of our approach.

arXiv commentA video of the rich set of learned behaviours can be found at https://youtu.be/mPKyvocNe_M

References in corpus (14)

• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Emergence of Locomotion Behaviours in Rich Environments
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Distributed Prioritized Experience Replay
• Learning Continuous Control Policies by Stochastic Value Gradients
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• Deep Successor Reinforcement Learning
• Sim2Real View Invariant Visual Servoing by Recurrent Control
• The Intentional Unintentional Agent: Learning to Solve Many Continuous Control Tasks Simultaneously
• Guided Policy Search as Approximate Mirror Descent

Cited by in corpus (80)

• An Introduction to Deep Reinforcement Learning
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Exploration in Deep Reinforcement Learning: A Survey
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
• Gradient Surgery for Multi-Task Learning
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Intelligent problem-solving as integrated hierarchical reinforcement learning
• A survey on intrinsic motivation in reinforcement learning
• Multi-Task Reinforcement Learning with Soft Modularization
• Meta reinforcement learning as task inference
• An empirical investigation of the challenges of real-world reinforcement learning
• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
• Safe Reinforcement Learning via Curriculum Induction
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• Relative Entropy Regularized Policy Iteration
• Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
• Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Unicorn: Continual Learning with a Universal, Off-policy Agent
• Goal-conditioned Imitation Learning
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• Many-Goals Reinforcement Learning
• Hierarchical Reinforcement Learning By Discovering Intrinsic Options
• Off-Policy Evaluation via Off-Policy Classification
• ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• Revisiting Design Choices in Proximal Policy Optimization
• Task-agnostic Exploration in Reinforcement Learning
• Learning and Planning with a Semantic Model
• Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study
• Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networks
• Learning Multi-Stage Tasks with One Demonstration via Self-Replay
• Data-efficient Hindsight Off-policy Option Learning
• Capability Iteration Network for Robot Path Planning
• Policy Search with Rare Significant Events: Choosing the Right Partner to Cooperate with
• Bayesian Relational Memory for Semantic Visual Navigation
• Lifelong Robotic Reinforcement Learning by Retaining Experiences
• Return-based Scaling: Yet Another Normalisation Trick for Deep RL
• TAAC: Temporally Abstract Actor-Critic for Continuous Control
• Action Guidance: Getting the Best of Sparse Rewards and Shaped Rewards for Real-time Strategy Games
• FOSS: A Self-Learned Doctor for Query Optimizer
• Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
• Dealing with Sparse Rewards Using Graph Neural Networks
• Discovery of Options via Meta-Learned Subgoals
• Hierarchical principles of embodied reinforcement learning: A review
• Predictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards
• Composite Q-learning: Multi-scale Q-function Decomposition and Separable Optimization
• Exploration via Hindsight Goal Generation
• Control What You Can: Intrinsically Motivated Task-Planning Agent
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Gap-Dependent Unsupervised Exploration for Reinforcement Learning
• PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning
• Curriculum Design for Teaching via Demonstrations: Theory and Applications
• Collaborative Behavior Models for Optimized Human-Robot Teamwork
• An Empowerment-based Solution to Robotic Manipulation Tasks with Sparse Rewards
• Curriculum Learning with a Progression Function
• Learning Compositional Neural Programs for Continuous Control
• Self-Paced Deep Reinforcement Learning
• Goal-constrained Sparse Reinforcement Learning for End-to-End Driving
• Flexible and Efficient Long-Range Planning Through Curious Exploration
• Reinforcement Learning for Vision-based Object Manipulation with Non-parametric Policy and Action Primitives
• Embedding Contextual Information through Reward Shaping in Multi-Agent Learning: A Case Study from Google Football
• Amanuensis: The Programmer's Apprentice
• Few-Shot Image-to-Semantics Translation for Policy Transfer in Reinforcement Learning
• Recursive Optimization of Finite Blocklength Allocation to Mitigate Age-of-Information Outage
• Parallelized Reverse Curriculum Generation
• HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem
• ACDER: Augmented Curiosity-Driven Experience Replay
• CubeTR: Learning to Solve The Rubiks Cube Using Transformers
• Training Transition Policies via Distribution Matching for Complex Tasks
• Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learning
• A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning
• Playing Atari Ball Games with Hierarchical Reinforcement Learning
• Learning Robotic Manipulation Tasks via Task Progress based Gaussian Reward and Loss Adjusted Exploration
• Explore and Control with Adversarial Surprise
• Learning in Sparse Rewards settings through Quality-Diversity algorithms