Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Making

Abstract

Many complicated real-world tasks can be broken down into smaller, more manageable parts, and planning with prior knowledge extracted from these simplified pieces is crucial for humans to make accurate decisions. However, replicating this process remains a challenge for AI agents and naturally raises two questions: How to extract discriminative knowledge representation from priors? How to develop a rational plan to decompose complex problems? Most existing representation learning methods employing a single encoder structure are fragile and sensitive to complex and diverse dynamics. To address this issue, we introduce a multiple-encoder and individual-predictor regime to learn task-essential representations from sufficient data for simple subtasks. Multiple encoders can extract adequate task-relevant dynamics without confusion, and the shared predictor can discriminate the task characteristics. We also use the attention mechanism to generate a top-k subtask planning tree, which customizes subtask execution plans in guiding complex decisions on unseen tasks. This process enables forward-looking and globality by flexibly adjusting the depth and width of the planning tree. Empirical results on a challenging platform composed of some basic simple tasks and combinatorially rich synthetic tasks consistently outperform some competitive baselines and demonstrate the benefits of our design.

Accepted by Machine Intelligence Research

References in corpus (24)

• Neural Machine Translation by Jointly Learning to Align and Translate
• On the Utility of Learning about Humans for Human-AI Coordination
• BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
• Sharing Knowledge in Multi-Task Deep Reinforcement Learning
• Multi-Task Reinforcement Learning with Soft Modularization
• A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning
• Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies
• Goal-Conditioned Reinforcement Learning with Imagined Subgoals
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning
• World Model as a Graph: Learning Latent Landmarks for Planning
• Multi-Task Reinforcement Learning with Context-based Representations
• Reward-Respecting Subtasks for Model-Based Reinforcement Learning
• Learning Neuro-Symbolic Skills for Bilevel Planning
• Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning
• FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization
• Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learning
• Learning Actionable Representations with Goal-Conditioned Policies
• Intention-Aware Navigation in Crowds with Extended-Space POMDP Planning
• Learning Robust State Abstractions for Hidden-Parameter Block MDPs
• Planning with Sequence Models through Iterative Energy Minimization
• Model-Based Offline Meta-Reinforcement Learning with Regularization
• Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning
• REMAX: Relational Representation for Multi-Agent Exploration