Scaling All-Goals Updates in Reinforcement Learning Using Convolutional Neural Networks
arXiv:1810.02927
Abstract
Being able to reach any desired location in the environment can be a valuable asset for an agent. Learning a policy to navigate between all pairs of states individually is often not feasible. An all-goals updating algorithm uses each transition to learn Q-values towards all goals simultaneously and off-policy. However the expensive numerous updates in parallel limited the approach to small tabular cases so far. To tackle this problem we propose to use convolutional network architectures to generate Q-values and updates for a large number of goals at once. We demonstrate the accuracy and generalization qualities of the proposed method on randomly generated mazes and Sokoban puzzles. In the case of on-screen goal coordinates the resulting mapping from frames to distance-maps directly informs the agent about which places are reachable and in how many steps. As an example of application we show that replacing the random actions in epsilon-greedy exploration by several actions towards feasible goals generates better exploratory trajectories on Montezuma's Revenge and Super Mario All-Stars games.
arXiv commentAAAI 2020, https://sites.google.com/view/q-map-rl
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Auto-Encoding Variational Bayes
- Continuous control with deep reinforcement learning
- Prioritized Experience Replay
- Dueling Network Architectures for Deep Reinforcement Learning
- Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
- Hindsight Experience Replay
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- Visual Reinforcement Learning with Imagined Goals
- Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
- Hybrid Reward Architecture for Reinforcement Learning
- Many-Goals Reinforcement Learning
- The Intentional Unintentional Agent: Learning to Solve Many Continuous Control Tasks Simultaneously
- Disentangling Controllable and Uncontrollable Factors of Variation by Interacting with the World
- Compression and Localization in Reinforcement Learning for ATARI Games