Active Reinforcement Learning with Monte-Carlo Tree Search
arXiv:1803.04926
Abstract
Active Reinforcement Learning (ARL) is a twist on RL where the agent observes reward information only if it pays a cost. This subtle change makes exploration substantially more challenging. Powerful principles in RL like optimism, Thompson sampling, and random exploration do not help with ARL. We relate ARL in tabular environments to Bayes-Adaptive MDPs. We provide an ARL algorithm using Monte-Carlo Tree Search that is asymptotically Bayes optimal. Experimentally, this algorithm is near-optimal on small Bandit problems and MDPs. On larger MDPs it outperforms a Q-learner augmented with specialised heuristics for ARL. By analysing exploration behaviour in detail, we uncover obstacles to scaling up simulation-based algorithms for ARL.
arXiv comment11 pages, 10 figures
References in corpus (8)
- Deep reinforcement learning from human preferences
- Generative Adversarial Imitation Learning
- Algorithms for multi-armed bandit problems
- Thinking Fast and Slow with Deep Learning and Tree Search
- Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
- Minimax Regret Bounds for Reinforcement Learning
- Robot Planning with Mathematical Models of Human State and Action
- Benchmarking for Bayesian Reinforcement Learning