Stochastic Neural Networks for Hierarchical Reinforcement Learning

Abstract

Deep reinforcement learning has achieved many impressive results in recent years. However, tasks with sparse rewards or long horizons continue to pose significant challenges. To tackle these important problems, we propose a general framework that first learns useful skills in a pre-training environment, and then leverages the acquired skills for learning faster in downstream tasks. Our approach brings together some of the strengths of intrinsic motivation and hierarchical methods: the learning of useful skill is guided by a single proxy reward, the design of which requires very minimal domain knowledge about the downstream tasks. Then a high-level policy is trained on top of these skills, providing a significant improvement of the exploration and allowing to tackle sparse rewards in the downstream tasks. To efficiently pre-train a large span of skills, we use Stochastic Neural Networks combined with an information-theoretic regularizer. Our experiments show that this combination is effective in learning a wide span of interpretable skills in a sample-efficient way, and can significantly boost the learning performance uniformly across a wide range of downstream tasks.

Published as a conference paper at ICLR 2017

References in corpus (2)

• DRAW: A Recurrent Neural Network For Image Generation
• Learning and Transfer of Modulated Locomotor Controllers

Cited by in corpus (61)

• On the Opportunities and Risks of Foundation Models
• An Introduction to Deep Reinforcement Learning
• Visual Reinforcement Learning with Imagined Goals
• Maximum a Posteriori Policy Optimisation
• Automatic Goal Generation for Reinforcement Learning Agents
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Eigenoption Discovery through the Deep Successor Representation
• Dynamics-Aware Unsupervised Discovery of Skills
• Latent Space Policies for Hierarchical Reinforcement Learning
• Multi-Level Discovery of Deep Options
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• Disentangling the independently controllable factors of variation by interacting with the world
• Routing Networks and the Challenges of Modular and Compositional Computation
• Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets
• Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning
• Learning Self-Imitating Diverse Policies
• Action and Perception as Divergence Minimization
• Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives
• From Pixels to Legs: Hierarchical Learning of Quadruped Locomotion
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Dynamics-aware Embeddings
• Meta-learning curiosity algorithms
• Adaptive trajectory-constrained exploration strategy for deep reinforcement learning
• Learning Pneumatic Non-Prehensile Manipulation with a Mobile Blower
• DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations
• Composing Task-Agnostic Policies with Deep Reinforcement Learning
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning
• Hierarchical Policy Learning is Sensitive to Goal Space Design
• Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck
• The Eigenoption-Critic Framework
• DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics
• Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation
• Influence-Based Multi-Agent Exploration
• Towards Autonomous Pipeline Inspection with Hierarchical Reinforcement Learning
• Hierarchical Learning for Modular Robots
• Multi-task Learning with Gradient Guided Policy Specialization
• Online Baum-Welch algorithm for Hierarchical Imitation Learning
• Disentangling causal effects for hierarchical reinforcement learning
• Adaptable Agent Populations via a Generative Model of Policies
• Inter-Level Cooperation in Hierarchical Reinforcement Learning
• Harnessing Distribution Ratio Estimators for Learning Agents with Quality and Diversity
• Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning
• Novel Policy Seeking with Constrained Optimization
• The Information Geometry of Unsupervised Reinforcement Learning
• Semantic RL with Action Grammars: Data-Efficient Learning of Hierarchical Task Abstractions
• Learning Diverse Policies with Soft Self-Generated Guidance
• Continual and Multi-task Reinforcement Learning With Shared Episodic Memory
• Hybrid system identification using switching density networks
• Temporal-adaptive Hierarchical Reinforcement Learning
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks
• Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning
• Transfer Learning by Modeling a Distribution over Policies
• Scalable, Decentralized Multi-Agent Reinforcement Learning Methods Inspired by Stigmergy and Ant Colonies
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms
• On Study of Mutual Information and its Estimation Methods
• Playing Atari Ball Games with Hierarchical Reinforcement Learning
• From proprioception to long-horizon planning in novel environments: A hierarchical RL model
• Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching