Stochastic Neural Networks for Hierarchical Reinforcement Learning

Abstract

Deep reinforcement learning has achieved many impressive results in recent years. However, tasks with sparse rewards or long horizons continue to pose significant challenges. To tackle these important problems, we propose a general framework that first learns useful skills in a pre-training environment, and then leverages the acquired skills for learning faster in downstream tasks. Our approach brings together some of the strengths of intrinsic motivation and hierarchical methods: the learning of useful skill is guided by a single proxy reward, the design of which requires very minimal domain knowledge about the downstream tasks. Then a high-level policy is trained on top of these skills, providing a significant improvement of the exploration and allowing to tackle sparse rewards in the downstream tasks. To efficiently pre-train a large span of skills, we use Stochastic Neural Networks combined with an information-theoretic regularizer. Our experiments show that this combination is effective in learning a wide span of interpretable skills in a sample-efficient way, and can significantly boost the learning performance uniformly across a wide range of downstream tasks.

arXiv commentPublished as a conference paper at ICLR 2017

References in corpus (11)

• Continuous control with deep reinforcement learning
• Categorical Reparameterization with Gumbel-Softmax
• DRAW: A Recurrent Neural Network For Image Generation
• Benchmarking Deep Reinforcement Learning for Continuous Control
• VIME: Variational Information Maximizing Exploration
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Unifying Count-Based Exploration and Intrinsic Motivation
• Learning and Transfer of Modulated Locomotor Controllers
• Generalization and Exploration via Randomized Value Functions
• Strategic Attentive Writer for Learning Macro-Actions
• On Multiplicative Integration with Recurrent Neural Networks

Cited by in corpus (63)

• On the Opportunities and Risks of Foundation Models
• An Introduction to Deep Reinforcement Learning
• Deep Reinforcement Learning: An Overview
• Reinforcement Learning with Deep Energy-Based Policies
• Visual Reinforcement Learning with Imagined Goals
• Maximum a Posteriori Policy Optimisation
• Automatic Goal Generation for Reinforcement Learning Agents
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Eigenoption Discovery through the Deep Successor Representation
• Dynamics-Aware Unsupervised Discovery of Skills
• Latent Space Policies for Hierarchical Reinforcement Learning
• Multi-Level Discovery of Deep Options
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• Disentangling the independently controllable factors of variation by interacting with the world
• Routing Networks and the Challenges of Modular and Compositional Computation
• Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets
• Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning
• Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives
• Learning Self-Imitating Diverse Policies
• Action and Perception as Divergence Minimization
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• From Pixels to Legs: Hierarchical Learning of Quadruped Locomotion
• Dynamics-aware Embeddings
• Meta-learning curiosity algorithms
• Adaptive trajectory-constrained exploration strategy for deep reinforcement learning
• Learning Pneumatic Non-Prehensile Manipulation with a Mobile Blower
• DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations
• Composing Task-Agnostic Policies with Deep Reinforcement Learning
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning
• Hierarchical Policy Learning is Sensitive to Goal Space Design
• Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck
• The Eigenoption-Critic Framework
• Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation
• DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics
• Influence-Based Multi-Agent Exploration
• Hierarchical Learning for Modular Robots
• Towards Autonomous Pipeline Inspection with Hierarchical Reinforcement Learning
• Online Baum-Welch algorithm for Hierarchical Imitation Learning
• Disentangling causal effects for hierarchical reinforcement learning
• Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning
• Inter-Level Cooperation in Hierarchical Reinforcement Learning
• Adaptable Agent Populations via a Generative Model of Policies
• Harnessing Distribution Ratio Estimators for Learning Agents with Quality and Diversity
• Multi-task Learning with Gradient Guided Policy Specialization
• The Information Geometry of Unsupervised Reinforcement Learning
• Learning Diverse Policies with Soft Self-Generated Guidance
• Novel Policy Seeking with Constrained Optimization
• Semantic RL with Action Grammars: Data-Efficient Learning of Hierarchical Task Abstractions
• Continual and Multi-task Reinforcement Learning With Shared Episodic Memory
• Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks
• Temporal-adaptive Hierarchical Reinforcement Learning
• Hybrid system identification using switching density networks
• From proprioception to long-horizon planning in novel environments: A hierarchical RL model
• Transfer Learning by Modeling a Distribution over Policies
• Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching
• Scalable, Decentralized Multi-Agent Reinforcement Learning Methods Inspired by Stigmergy and Ant Colonies
• Playing Atari Ball Games with Hierarchical Reinforcement Learning
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms
• On Study of Mutual Information and its Estimation Methods