Learning Abstract Models for Strategic Exploration and Fast Reward Transfer

Abstract

Model-based reinforcement learning (RL) is appealing because (i) it enables planning and thus more strategic exploration, and (ii) by decoupling dynamics from rewards, it enables fast transfer to new reward functions. However, learning an accurate Markov Decision Process (MDP) over high-dimensional states (e.g., raw pixels) is extremely challenging because it requires function approximation, which leads to compounding errors. Instead, to avoid compounding errors, we propose learning an abstract MDP over abstract states: low-dimensional coarse representations of the state (e.g., capturing agent position, ignoring other objects). We assume access to an abstraction function that maps the concrete states to abstract states. In our approach, we construct an abstract MDP, which grows through strategic exploration via planning. Similar to hierarchical RL approaches, the abstract actions of the abstract MDP are backed by learned subpolicies that navigate between abstract states. Our approach achieves strong results on three of the hardest Arcade Learning Environment games (Montezuma's Revenge, Pitfall!, and Private Eye), including superhuman performance on Pitfall! without demonstrations. After training on one task, we can reuse the learned abstract MDP for new reward functions, achieving higher reward in 1000x fewer samples than model-free methods trained from scratch.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Overcoming catastrophic forgetting in neural networks
• Dueling Network Architectures for Deep Reinforcement Learning
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Exploration by Random Network Distillation
• Go-Explore: a New Approach for Hard-Exploration Problems
• Recurrent Environment Simulators
• Observe and Look Further: Achieving Consistent Performance on Atari
• State Abstraction in MAXQ Hierarchical Reinforcement Learning
• Deep Abstract Q-Networks

Cited by in corpus (2)

• Exploration in Deep Reinforcement Learning: A Survey
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices