Temporally Abstract Partial Models
arXiv:2108.03213
Abstract
Humans and animals have the ability to reason and make predictions about different courses of action at many time scales. In reinforcement learning, option models (Sutton, Precup \& Singh, 1999; Precup, 2000) provide the framework for this kind of temporally abstract prediction and reasoning. Natural intelligent agents are also able to focus their attention on courses of action that are relevant or feasible in a given situation, sometimes termed affordable actions. In this paper, we define a notion of affordances for options, and develop temporally abstract partial option models, that take into account the fact that an option might be affordable only in certain situations. We analyze the trade-offs between estimation and approximation error in planning and learning when using such models, and identify some interesting special cases. Additionally, we demonstrate empirically the potential impact of partial option models on the efficiency of planning.
arXiv comment34 pages, 5 figures
References in corpus (8)
- COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
- Neural Predictive Belief Representations
- What can I do here? A Theory of Affordances in Reinforcement Learning
- Self-Correcting Models for Model-Based Reinforcement Learning
- What can you do with a rock? Affordance extraction via word embeddings
- Learning Awareness Models
- Deep Affordance Foresight: Planning Through What Can Be Done in the Future
- Launchpad: A Programming Model for Distributed Machine Learning Research