From proprioception to long-horizon planning in novel environments: A hierarchical RL model

Abstract

For an intelligent agent to flexibly and efficiently operate in complex environments, they must be able to reason at multiple levels of temporal, spatial, and conceptual abstraction. At the lower levels, the agent must interpret their proprioceptive inputs and control their muscles, and at the higher levels, the agent must select goals and plan how they will achieve those goals. It is clear that each of these types of reasoning is amenable to different types of representations, algorithms, and inputs. In this work, we introduce a simple, three-level hierarchical architecture that reflects these distinctions. The low-level controller operates on the continuous proprioceptive inputs, using model-free learning to acquire useful behaviors. These in turn induce a set of mid-level dynamics, which are learned by the mid-level controller and used for model-predictive control, to select a behavior to activate at each timestep. The high-level controller leverages a discrete, graph representation for goal selection and path planning to specify targets for the mid-level controller. We apply our method to a series of navigation tasks in the Mujoco Ant environment, consistently demonstrating significant improvements in sample-efficiency compared to prior model-free, model-based, and hierarchical RL methods. Finally, as an illustrative example of the advantages of our architecture, we apply our method to a complex maze environment that requires efficient exploration and long-horizon planning.

References in corpus (16)

• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• VIME: Variational Information Maximizing Exploration
• Benchmarking Model-Based Reinforcement Learning
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Variational Intrinsic Control
• Learning and Transfer of Modulated Locomotor Controllers
• Variational Option Discovery Algorithms
• Dynamics-Aware Unsupervised Discovery of Skills
• Latent Space Policies for Hierarchical Reinforcement Learning
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Semi-parametric Topological Memory for Navigation
• Disentangling the independently controllable factors of variation by interacting with the world
• Search on the Replay Buffer: Bridging Planning and Reinforcement Learning