Outcome-Driven Reinforcement Learning via Variational Inference
arXiv:2104.10190
Abstract
While reinforcement learning algorithms provide automated acquisition of optimal policies, practical application of such methods requires a number of design decisions, such as manually designing reward functions that not only define the task, but also provide sufficient shaping to accomplish it. In this paper, we view reinforcement learning as inferring policies that achieve desired outcomes, rather than as a problem of maximizing rewards. To solve this inference problem, we establish a novel variational inference formulation that allows us to derive a well-shaped reward function which can be learned directly from environment interactions. From the corresponding variational objective, we also derive a new probabilistic Bellman backup operator and use it to develop an off-policy algorithm to solve goal-directed tasks. We empirically demonstrate that this method eliminates the need to hand-craft reward functions for a suite of diverse manipulation and locomotion tasks and leads to effective goal-directed behaviors.
Published in Advances in Neural Information Processing Systems 34 (NeurIPS 2021)
References in corpus (9)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Soft Actor-Critic Algorithms and Applications
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
- Visual Reinforcement Learning with Imagined Goals
- Data-Efficient Hierarchical Reinforcement Learning
- Temporal Difference Models: Model-Free Deep RL for Model-Based Control
- VIREL: A Variational Inference Framework for Reinforcement Learning
- Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning
- Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning