Learning Montezuma's Revenge from a Single Demonstration

Abstract

We propose a new method for learning from a single demonstration to solve hard exploration tasks like the Atari game Montezuma's Revenge. Instead of imitating human demonstrations, as proposed in other recent works, our approach is to maximize rewards directly. Our agent is trained using off-the-shelf reinforcement learning, but starts every episode by resetting to a state from a demonstration. By starting from such demonstration states, the agent requires much less exploration to learn a game compared to when it starts from the beginning of the game at every episode. We analyze reinforcement learning for tasks with sparse rewards in a simple toy environment, where we show that the run-time of standard RL methods scales exponentially in the number of states between rewards. Our method reduces this to quadratic scaling, opening up many tasks that were previously infeasible. We then apply our method to Montezuma's Revenge, for which we present a trained agent achieving a high-score of 74,500, better than any previously published result.

arXiv commentDeep RL Workshop, NeurIPS 2018

References in corpus (17)

• WaveNet: A Generative Model for Raw Audio
• Prioritized Experience Replay
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Count-Based Exploration with Neural Density Models
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Model-Based Bayesian Exploration
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Reverse Curriculum Generation for Reinforcement Learning
• Combining policy gradient and Q-learning
• Observe and Look Further: Achieving Consistent Performance on Atari
• UCB Exploration via Q-Ensembles
• Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay
• Backplay: "Man muss immer umkehren"
• Expert-augmented actor-critic for ViZDoom and Montezumas Revenge

Cited by in corpus (40)

• Exploration in Deep Reinforcement Learning: A Survey
• Go-Explore: a New Approach for Hard-Exploration Problems
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• The NetHack Learning Environment
• Generative Personas That Behave and Experience Like Humans
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Model-based Reinforcement Learning for Predictions and Control for Limit Order Books
• Show Us the Way: Learning to Manage Dialog from Demonstrations
• Imitation Learning: Progress, Taxonomies and Challenges
• ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrations
• Using Monte Carlo Tree Search as a Demonstrator within Asynchronous Deep RL
• Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning
• Expert-augmented actor-critic for ViZDoom and Montezumas Revenge
• Meta Automatic Curriculum Learning
• Deep Reinforcement Learning in Fluid Mechanics: a promising method for both Active Flow Control and Shape Optimization
• TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL
• HiER: Highlight Experience Replay for Boosting Off-Policy Reinforcement Learning Agents
• Predictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards
• Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning
• PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning
• Exploring Restart Distributions
• Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimally
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Continuous Control with Action Quantization from Demonstrations
• Plan-Based Relaxed Reward Shaping for Goal-Directed Tasks
• Evolutionary Selective Imitation: Interpretable Agents by Imitation Learning Without a Demonstrator
• Curriculum learning for multilevel budgeted combinatorial problems
• Guided Exploration with Proximal Policy Optimization using a Single Demonstration
• Diluted Near-Optimal Expert Demonstrations for Guiding Dialogue Stochastic Policy Optimisation
• Adaptive Leader-Follower Formation Control and Obstacle Avoidance via Deep Reinforcement Learning
• Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation
• Reparameterized Variational Divergence Minimization for Stable Imitation
• Lucid Dreaming for Experience Replay: Refreshing Past States with the Current Policy
• Automatic Curricula via Expert Demonstrations
• Reinforcement Learning via Reasoning from Demonstration
• Deep Reinforcement Learning for Complex Manipulation Tasks with Sparse Feedback
• Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration
• Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning