Exploiting Hierarchy for Learning and Transfer in KL-regularized RL

Abstract

As reinforcement learning agents are tasked with solving more challenging and diverse tasks, the ability to incorporate prior knowledge into the learning system and to exploit reusable structure in solution space is likely to become increasingly important. The KL-regularized expected reward objective constitutes one possible tool to this end. It introduces an additional component, a default or prior behavior, which can be learned alongside the policy and as such partially transforms the reinforcement learning problem into one of behavior modelling. In this work we consider the implications of this framework in cases where both the policy and default behavior are augmented with latent variables. We discuss how the resulting hierarchical structures can be used to implement different inductive biases and how their modularity can benefit transfer. Empirically we find that they can lead to faster learning and transfer on a range of continuous control tasks.

References in corpus (26)

• Continuous control with deep reinforcement learning
• Neural Architecture Search with Reinforcement Learning
• Trust Region Policy Optimization
• Emergence of Locomotion Behaviours in Rich Environments
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Reinforcement Learning with Deep Energy-Based Policies
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• Learning Continuous Control Policies by Stochastic Value Gradients
• Optimal control as a graphical model inference problem
• Thermodynamics as a theory of decision-making with information processing costs
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• FeUdal Networks for Hierarchical Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Variational Intrinsic Control
• Synthesizing Programs for Images using Reinforced Adversarial Learning
• Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
• Learning and Transfer of Modulated Locomotor Controllers
• Neural probabilistic motor primitives for humanoid control
• Policy Distillation
• Multi-Level Discovery of Deep Options
• Kickstarting Deep Reinforcement Learning
• Learning to Share and Hide Intentions using Information Regularization
• Information asymmetry in KL-regularized RL
• A Unified Bellman Equation for Causal Information and Value in Markov Decision Processes

Cited by in corpus (12)

• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning
• Action and Perception as Divergence Minimization
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• Deep RL With Information Constrained Policies: Generalization in Continuous Control
• Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity
• Iterative Amortized Policy Optimization
• Goal Kernel Planning: Linearly-Solvable Non-Markovian Policies for Logical Tasks with Goal-Conditioned Options
• An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning
• Reparameterized Variational Divergence Minimization for Stable Imitation