Self-Consistent Models and Values

Abstract

Learned models of the environment provide reinforcement learning (RL) agents with flexible ways of making predictions about the environment. In particular, models enable planning, i.e. using more computation to improve value functions or policies, without requiring additional environment interactions. In this work, we investigate a way of augmenting model-based RL, by additionally encouraging a learned model and value function to be jointly \emph{self-consistent}. Our approach differs from classic planning methods such as Dyna, which only update values to be consistent with the model. We propose multiple self-consistency updates, evaluate these in both tabular and function approximation settings, and find that, with appropriate choices, self-consistency helps both policy evaluation and control.

arXiv commentNeurIPS 2021

References in corpus (19)

• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Imagination-Augmented Agents for Deep Reinforcement Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• Understanding Back-Translation at Scale
• When to use parametric models in reinforcement learning?
• Observe and Look Further: Achieving Consistent Performance on Atari
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Value Prediction Network
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces
• TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning
• Causally Correct Partial Models for Reinforcement Learning
• Shaping Belief States with Generative Environment Models for RL
• Podracer architectures for scalable Reinforcement Learning
• PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• Value-driven Hindsight Modelling
• Frequency-based Search-control in Dyna