Backplay: "Man muss immer umkehren"

Abstract

Model-free reinforcement learning (RL) requires a large number of trials to learn a good policy, especially in environments with sparse rewards. We explore a method to improve the sample efficiency when we have access to demonstrations. Our approach, Backplay, uses a single demonstration to construct a curriculum for a given task. Rather than starting each training episode in the environment's fixed initial state, we start the agent near the end of the demonstration and move the starting point backwards during the course of training until we reach the initial state. Our contributions are that we analytically characterize the types of environments where Backplay can improve training speed, demonstrate the effectiveness of Backplay both in large grid worlds and a complex four player zero-sum game (Pommerman), and show that Backplay compares favorably to other competitive methods known to improve sample efficiency. This includes reward shaping, behavioral cloning, and reverse curriculum generation.

arXiv commentAAAI-19 Workshop on Reinforcement Learning in Games; 0xd1a80a702b8170f6abeaabcf32a0c4c4401e9177

References in corpus (17)

• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• Deep Reinforcement Learning for Dialogue Generation
• Deep Q-learning from Demonstrations
• Reverse Curriculum Generation for Reinforcement Learning
• Query-Efficient Imitation Learning for End-to-End Autonomous Driving
• Maintaining cooperation in complex social dilemmas using deep reinforcement learning
• Safe and Nested Subgame Solving for Imperfect-Information Games
• Reinforcement Learning from Imperfect Demonstrations
• Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay
• Kickstarting Deep Reinforcement Learning
• Recall Traces: Backtracking Models for Efficient Reinforcement Learning
• Consequentialist conditional cooperation in social dilemmas with imperfect information
• Solving the Rubik's Cube Without Human Knowledge
• Forward-Backward Reinforcement Learning
• On the mixing time and spectral gap for birth and death chains

Cited by in corpus (15)

• Go-Explore: a New Approach for Hard-Exploration Problems
• Learning Montezuma's Revenge from a Single Demonstration
• Interactive Fiction Games: A Colossal Adventure
• Real-time tree search with pessimistic scenarios
• Using Monte Carlo Tree Search as a Demonstrator within Asynchronous Deep RL
• Accelerating Training in Pommerman with Imitation and Reinforcement Learning
• Safer Deep RL with Shallow MCTS: A Case Study in Pommerman
• PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning
• Competitive Experience Replay
• REMAX: Relational Representation for Multi-Agent Exploration
• ABC-LMPC: Safe Sample-Based Learning MPC for Stochastic Nonlinear Dynamical Systems with Adjustable Boundary Conditions
• Automated curriculum generation for Policy Gradients from Demonstrations
• Generative Exploration and Exploitation
• Automatic Curricula via Expert Demonstrations
• Lucid Dreaming for Experience Replay: Refreshing Past States with the Current Policy