FeUdal Networks for Hierarchical Reinforcement Learning

Abstract

We introduce FeUdal Networks (FuNs): a novel architecture for hierarchical reinforcement learning. Our approach is inspired by the feudal reinforcement learning proposal of Dayan and Hinton, and gains power and efficacy by decoupling end-to-end learning across multiple levels -- allowing it to utilise different resolutions of time. Our framework employs a Manager module and a Worker module. The Manager operates at a lower temporal resolution and sets abstract goals which are conveyed to and enacted by the Worker. The Worker generates primitive actions at every tick of the environment. The decoupled structure of FuN conveys several benefits -- in addition to facilitating very long timescale credit assignment it also encourages the emergence of sub-policies associated with different goals set by the Manager. These properties allow FuN to dramatically outperform a strong baseline agent on tasks that involve long-term credit assignment or memorisation. We demonstrate the performance of our proposed system on a range of tasks from the ATARI suite and also from a 3D DeepMind Lab environment.

Cited by in corpus (86)

• A Brief Survey of Deep Reinforcement Learning
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Intelligent problem-solving as integrated hierarchical reinforcement learning
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Competitive Gradient Descent
• CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning
• Search on the Replay Buffer: Bridging Planning and Reinforcement Learning
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
• BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments
• Beyond spiking networks: the computational advantages of dendritic amplification and input segregation
• Hierarchical Reinforcement Learning By Discovering Intrinsic Options
• Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning
• Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization
• Learning Compositional Neural Programs with Recursive Tree Search and Planning
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• Deep Reinforcement Learning and Transportation Research: A Comprehensive Review
• Reinforcement Learning for Multi-Product Multi-Node Inventory Management in Supply Chains
• Dynamics-aware Embeddings
• Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks
• Sub-policy Adaptation for Hierarchical Reinforcement Learning
• HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators
• Hierarchical Policy Learning is Sensitive to Goal Space Design
• Options as responses: Grounding behavioural hierarchies in multi-agent RL
• Solving Compositional Reinforcement Learning Problems via Task Reduction
• Learning Hierarchical Teaching Policies for Cooperative Agents
• Long-Term Planning and Situational Awareness in OpenAI Five
• On Reinforcement Learning for Full-length Game of StarCraft
• TAAC: Temporally Abstract Actor-Critic for Continuous Control
• SocialInteractionGAN: Multi-person Interaction Sequence Generation
• Diversity-Enriched Option-Critic
• Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations
• Learning sparse representations in reinforcement learning
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives
• Hierarchical principles of embodied reinforcement learning: A review
• Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving
• Dilated Convolution with Dilated GRU for Music Source Separation
• Model Primitive Hierarchical Lifelong Reinforcement Learning
• Provable Hierarchical Imitation Learning via EM
• Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity
• Towards Autonomous Pipeline Inspection with Hierarchical Reinforcement Learning
• Thalamocortical motor circuit insights for more robust hierarchical control of complex sequences
• Hierarchical reinforcement learning for efficient exploration and transfer
• Correcting Experience Replay for Multi-Agent Communication
• Hierarchies of Planning and Reinforcement Learning for Robot Navigation
• Danger-aware Adaptive Composition of DRL Agents for Self-navigation
• Stochastic Extragradient: General Analysis and Improved Rates
• From semantics to execution: Integrating action planning with reinforcement learning for robotic causal problem-solving
• Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
• Action Redundancy in Reinforcement Learning
• Hierarchical Reinforcement Learning for Deep Goal Reasoning: An Expressiveness Analysis
• Self-supervised Reinforcement Learning with Independently Controllable Subgoals
• Deciding What to Learn: A Rate-Distortion Approach
• Plan Arithmetic: Compositional Plan Vectors for Multi-Task Control
• Hierarchical Skills for Efficient Exploration
• Learning the Solution Manifold in Optimization and Its Application in Motion Planning
• TTR-Based Reward for Reinforcement Learning with Implicit Model Priors
• Object-oriented state editing for HRL
• Perception-Prediction-Reaction Agents for Deep Reinforcement Learning
• Planning in Hierarchical Reinforcement Learning: Guarantees for Using Local Policies
• Subgoal-based Reward Shaping to Improve Efficiency in Reinforcement Learning
• Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation
• Hierarchical Reinforcement Learning with Optimal Level Synchronization Based on Flow-Based Deep Generative Model
• Scalable, Decentralized Multi-Agent Reinforcement Learning Methods Inspired by Stigmergy and Ant Colonies
• Spatially and Seamlessly Hierarchical Reinforcement Learning for State Space and Policy space in Autonomous Driving
• Delayed Rewards Calibration via Reward Empirical Sufficiency
• Active Hierarchical Imitation and Reinforcement Learning
• Feudal Reinforcement Learning by Reading Manuals
• Reward Shaping with Dynamic Trajectory Aggregation
• Reinforcement Learning via Reasoning from Demonstration
• Follow the Neurally-Perturbed Leader for Adversarial Training
• Developing cooperative policies for multi-stage tasks
• Hierarchical Neural Dynamic Policies
• Provable Hierarchy-Based Meta-Reinforcement Learning
• Playing Atari Ball Games with Hierarchical Reinforcement Learning
• Generalization in Text-based Games via Hierarchical Reinforcement Learning
• Weakly Supervised Video Summarization by Hierarchical Reinforcement Learning
• Learning Memory-Dependent Continuous Control from Demonstrations