Forethought and Hindsight in Credit Assignment

Abstract

We address the problem of credit assignment in reinforcement learning and explore fundamental questions regarding the way in which an agent can best use additional computation to propagate new information, by planning with internal models of the world to improve its predictions. Particularly, we work to understand the gains and peculiarities of planning employed as forethought via forward models or as hindsight operating with backward models. We establish the relative merits, limitations and complementary properties of both planning mechanisms in carefully constructed scenarios. Further, we investigate the best use of models in planning, primarily focusing on the selection of states in which predictions should be (re)-evaluated. Lastly, we discuss the issue of model estimation and highlight a spectrum of methods that stretch from explicit environment-dynamics predictors to more abstract planner-aware models.

References in corpus (13)

• Gaussian Processes for Data-Efficient Learning in Robotics and Control
• Recurrent World Models Facilitate Policy Evolution
• The Predictron: End-To-End Learning and Planning
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping
• Discovering Reinforcement Learning Algorithms
• Consistent On-Line Off-Policy Evaluation
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Planning by Prioritized Sweeping with Small Backups
• Emphatic Temporal-Difference Learning
• The Termination Critic
• Generative predecessor models for sample-efficient imitation learning
• Planning with Expectation Models

Cited by in corpus (2)

• Offline Reinforcement Learning with Reverse Model-based Imagination
• Learning Retrospective Knowledge with Reverse Reinforcement Learning