Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay

Abstract

This paper introduces a novel method for learning how to play the most difficult Atari 2600 games from the Arcade Learning Environment using deep reinforcement learning. The proposed method, human checkpoint replay, consists in using checkpoints sampled from human gameplay as starting points for the learning process. This is meant to compensate for the difficulties of current exploration strategies, such as epsilon-greedy, to find successful control policies in games with sparse rewards. Like other deep reinforcement learning architectures, our model uses a convolutional neural network that receives only raw pixel inputs to estimate the state value function. We tested our method on Montezuma's Revenge and Private Eye, two of the most challenging games from the Atari platform. The results we obtained show a substantial improvement compared to previous learning approaches, as well as over a random player. We also propose a method for training deep reinforcement learning agents using human gameplay experience, which we call human experience replay.

6 pages, 2 figures, EGPAI 2016 - Evaluating General Purpose AI, workshop held in conjunction with ECAI 2016

References in corpus (5)

• Playing Atari with Deep Reinforcement Learning
• Prioritized Experience Replay
• Deep Exploration via Bootstrapped DQN
• Massively Parallel Methods for Deep Reinforcement Learning
• Planning by Prioritized Sweeping with Small Backups

Cited by in corpus (18)

• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey
• Learning Montezuma's Revenge from a Single Demonstration
• Active Deep Q-learning with Demonstration
• Backplay: "Man muss immer umkehren"
• The Atari Grand Challenge Dataset
• Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
• Sample-Efficient Learning of Nonprehensile Manipulation Policies via Physics-Based Informed State Distributions
• Neural-encoding Human Experts' Domain Knowledge to Warm Start Reinforcement Learning
• Learning hierarchical behavior and motion planning for autonomous driving
• Learning Multi-agent Skills for Tabular Reinforcement Learning using Factor Graphs
• Hybrid Reinforcement Learning with Expert State Sequences
• PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning
• Baconian: A Unified Open-source Framework for Model-Based Reinforcement Learning
• Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations
• Which Channel to Ask My Question? Personalized Customer Service Request Stream Routing using Deep Reinforcement Learning
• Lucid Dreaming for Experience Replay: Refreshing Past States with the Current Policy
• Reinforcement Learning For Constraint Satisfaction Game Agents (15-Puzzle, Minesweeper, 2048, and Sudoku)
• Learning to Shape Rewards using a Game of Two Partners