Reusability and Transferability of Macro Actions for Reinforcement Learning
arXiv:1908.01478 · doi:10.1145/3514260
Abstract
Conventional reinforcement learning (RL) typically determines an appropriate primitive action at each timestep. However, by using a proper macro action, defined as a sequence of primitive actions, an agent is able to bypass intermediate states to a farther state and facilitate its learning procedure. The problem we would like to investigate is what associated beneficial properties that macro actions may possess. In this paper, we unveil the properties of reusability and transferability of macro actions. The first property, reusability, means that a macro action generated along with one RL method can be reused by another RL method for training, while the second one, transferability, means that a macro action can be utilized for training agents in similar environments with different reward settings. In our experiments, we first generate macro actions along with RL methods. We then provide a set of analyses to reveal the properties of reusability and transferability of the generated macro actions.
References in corpus (12)
- Playing Atari with Deep Reinforcement Learning
- The Arcade Learning Environment: An Evaluation Platform for General Agents
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
- Large-Scale Study of Curiosity-Driven Learning
- Evolutionary Algorithms for Reinforcement Learning
- Unifying Count-Based Exploration and Intrinsic Motivation
- Macro-FF: Improving AI Planning with Automatically Learned Macro-Operators
- Accelerated Methods for Deep Reinforcement Learning
- Marvin: A Heuristic Search Planner with Online Macro-Action Learning
- Deep Reinforcement Learning With Macro-Actions
- Macro Action Reinforcement Learning with Sequence Disentanglement using Variational Autoencoder