Behavior Priors for Efficient Reinforcement Learning

Abstract

As we deploy reinforcement learning agents to solve increasingly challenging problems, methods that allow us to inject prior knowledge about the structure of the world and effective solution strategies becomes increasingly important. In this work we consider how information and architectural constraints can be combined with ideas from the probabilistic modeling literature to learn behavior priors that capture the common movement and interaction patterns that are shared across a set of related tasks or contexts. For example the day-to day behavior of humans comprises distinctive locomotion and manipulation patterns that recur across many different situations and goals. We discuss how such behavior patterns can be captured using probabilistic trajectory models and how these can be integrated effectively into reinforcement learning schemes, e.g.\ to facilitate multi-task and transfer learning. We then extend these ideas to latent variable models and consider a formulation to learn hierarchical priors that capture different aspects of the behavior in reusable modules. We discuss how such latent variable formulations connect to related work on hierarchical reinforcement learning (HRL) and mutual information and curiosity based objectives, thereby offering an alternative perspective on existing ideas. We demonstrate the effectiveness of our framework by applying it to a range of simulated continuous control domains.

Submitted to Journal of Machine Learning Research (JMLR)

References in corpus (31)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• A Simple Neural Attentive Meta-Learner
• A Recurrent Latent Variable Model for Sequential Data
• Emergence of Locomotion Behaviours in Rich Environments
• DeepMind Control Suite
• Reinforcement Learning with Deep Energy-Based Policies
• Deep Variational Information Bottleneck
• Optimal control as a graphical model inference problem
• Learning Dexterous In-Hand Manipulation
• Thermodynamics as a theory of decision-making with information processing costs
• Data-Efficient Hierarchical Reinforcement Learning
• Composing graphical models with neural networks for structured representations and fast inference
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Learning and Transfer of Modulated Locomotor Controllers
• Neural probabilistic motor primitives for humanoid control
• Multi-Level Discovery of Deep Options
• Meta reinforcement learning as task inference
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• Fixing a Broken ELBO
• Learning to Share and Hide Intentions using Information Regularization
• Information asymmetry in KL-regularized RL
• A Unified Bellman Equation for Causal Information and Value in Markov Decision Processes
• Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
• Guided Policy Search as Approximate Mirror Descent
• Path Integral Guided Policy Search

Cited by in corpus (4)

• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
• Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models
• A Simple Approach to Continual Learning by Transferring Skill Parameters
• Reward is enough for convex MDPs