Developing cooperative policies for multi-stage tasks

Abstract

This paper proposes the Cooperative Soft Actor Critic (CSAC) method of enabling consecutive reinforcement learning agents to cooperatively solve a long time horizon multi-stage task. This method is achieved by modifying the policy of each agent to maximise both the current and next agent's critic. Cooperatively maximising each agent's critic allows each agent to take actions that are beneficial for its task as well as subsequent tasks. Using this method in a multi-room maze domain, the cooperative policies were able to outperform both uncooperative policies as well as a single agent trained across the entire domain. CSAC achieved a success rate of at least 20\% higher than the uncooperative policies, and converged on a solution at least 4 times faster than the single agent.

This paper was submitted to RA-L on June 20th 2020

References in corpus (10)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• FeUdal Networks for Hierarchical Reinforcement Learning
• Data-Efficient Hierarchical Reinforcement Learning
• Modular Multitask Reinforcement Learning with Policy Sketches
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Meta Learning Shared Hierarchies
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Latent Space Policies for Hierarchical Reinforcement Learning
• HIRL: Hierarchical Inverse Reinforcement Learning for Long-Horizon Tasks with Delayed Rewards