Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey

Abstract

Reinforcement learning (RL) is a popular paradigm for addressing sequential decision tasks in which the agent has only limited environmental feedback. Despite many advances over the past three decades, learning in many domains still requires a large amount of interaction with the environment, which can be prohibitively expensive in realistic scenarios. To address this problem, transfer learning has been applied to reinforcement learning such that experience gained in one task can be leveraged when starting to learn the next, harder task. More recently, several lines of research have explored how tasks, or data samples themselves, can be sequenced into a curriculum for the purpose of learning a problem that may otherwise be too difficult to learn from scratch. In this article, we present a framework for curriculum learning (CL) in reinforcement learning, and use it to survey and classify existing CL methods in terms of their assumptions, capabilities, and goals. Finally, we use our framework to find open problems and suggest directions for future RL curriculum learning research.

References in corpus (19)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Robust Adversarial Reinforcement Learning
• Emergent Tool Use From Multi-Agent Autocurricula
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks
• Interactive Learning from Policy-Dependent Human Feedback
• Strategic Attentive Writer for Learning Macro-Actions
• Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay
• CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning
• Transfer from Multiple MDPs
• ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks
• Data-Efficient Policy Evaluation Through Behavior Policy Search
• Theory of Curriculum Learning, with Convex Loss Functions
• Evolutionarily-Curated Curriculum Learning for Deep Reinforcement Learning Agents
• A gray-box approach for curriculum learning
• Faster Reinforcement Learning Using Active Simulators
• Curriculum Learning with a Progression Function

Cited by in corpus (25)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Sim-To-Real Transfer for Visual Reinforcement Learning of Deformable Object Manipulation for Robot-Assisted Surgery
• Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning
• Curriculum-based Reinforcement Learning for Distribution System Critical Load Restoration
• Bridging adaptive management and reinforcement learning for more robust decisions
• Knowledge Transfer for Cross-Domain Reinforcement Learning: A Systematic Review
• A new hope for network model generalization
• A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems
• A Survey of Reinforcement Learning for Optimization in Automation
• RL-LABEL: A Deep Reinforcement Learning Approach Intended for AR Label Placement in Dynamic Scenarios
• UAV-based path planning for efficient localization of non-uniformly distributed weeds using prior knowledge: A reinforcement-learning approach
• Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
• Leveraging a realistic synthetic database to learn Shape-from-Shading for estimating the colon depth in colonoscopy images
• On the Benefit of Optimal Transport for Curriculum Reinforcement Learning
• Guided Curriculum Learning for Walking Over Complex Terrain
• Do Data-based Curricula Work?
• Real-Time Model-Free Deep Reinforcement Learning for Force Control of a Series Elastic Actuator
• Dealing with Sparse Rewards Using Graph Neural Networks
• Reinforcement Learning with Success Induced Task Prioritization
• Reinforcement Learning with Subspaces using Free Energy Paradigm
• On the Statistical Benefits of Curriculum Learning
• Co-Training an Observer and an Evading Target
• Curriculum-Based Imitation of Versatile Skills
• GloCAL: Glocalized Curriculum-Aided Learning of Multiple Tasks with Application to Robotic Grasping