FeUdal Networks for Hierarchical Reinforcement Learning

Abstract

We introduce FeUdal Networks (FuNs): a novel architecture for hierarchical reinforcement learning. Our approach is inspired by the feudal reinforcement learning proposal of Dayan and Hinton, and gains power and efficacy by decoupling end-to-end learning across multiple levels -- allowing it to utilise different resolutions of time. Our framework employs a Manager module and a Worker module. The Manager operates at a lower temporal resolution and sets abstract goals which are conveyed to and enacted by the Worker. The Worker generates primitive actions at every tick of the environment. The decoupled structure of FuN conveys several benefits -- in addition to facilitating very long timescale credit assignment it also encourages the emergence of sub-policies associated with different goals set by the Manager. These properties allow FuN to dramatically outperform a strong baseline agent on tasks that involve long-term credit assignment or memorisation. We demonstrate the performance of our proposed system on a range of tasks from the ATARI suite and also from a 3D DeepMind Lab environment.

Cited by in corpus (38)

• A Brief Survey of Deep Reinforcement Learning
• Intelligent problem-solving as integrated hierarchical reinforcement learning
• Search on the Replay Buffer: Bridging Planning and Reinforcement Learning
• BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning
• Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization
• Reinforcement Learning for Multi-Product Multi-Node Inventory Management in Supply Chains
• Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks
• HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators
• Hierarchical Policy Learning is Sensitive to Goal Space Design
• Solving Compositional Reinforcement Learning Problems via Task Reduction
• Long-Term Planning and Situational Awareness in OpenAI Five
• Dilated Convolution with Dilated GRU for Music Source Separation
• Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving
• Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Thalamocortical motor circuit insights for more robust hierarchical control of complex sequences
• Model Primitive Hierarchical Lifelong Reinforcement Learning
• Hierarchies of Planning and Reinforcement Learning for Robot Navigation
• Hierarchical reinforcement learning for efficient exploration and transfer
• Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
• Perception-Prediction-Reaction Agents for Deep Reinforcement Learning
• Hierarchical Reinforcement Learning for Deep Goal Reasoning: An Expressiveness Analysis
• Object-oriented state editing for HRL
• Learning the Solution Manifold in Optimization and Its Application in Motion Planning
• Hierarchical Skills for Efficient Exploration
• Hierarchical Neural Dynamic Policies
• Playing Atari Ball Games with Hierarchical Reinforcement Learning
• Reinforcement Learning via Reasoning from Demonstration
• Provable Hierarchy-Based Meta-Reinforcement Learning
• Feudal Reinforcement Learning by Reading Manuals
• Generalization in Text-based Games via Hierarchical Reinforcement Learning
• Learning Memory-Dependent Continuous Control from Demonstrations
• Weakly Supervised Video Summarization by Hierarchical Reinforcement Learning
• Reward Shaping with Dynamic Trajectory Aggregation
• Developing cooperative policies for multi-stage tasks
• Active Hierarchical Imitation and Reinforcement Learning