Behaviour-conditioned policies for cooperative reinforcement learning tasks

Abstract

The cooperation among AI systems, and between AI systems and humans is becoming increasingly important. In various real-world tasks, an agent needs to cooperate with unknown partner agent types. This requires the agent to assess the behaviour of the partner agent during a cooperative task and to adjust its own policy to support the cooperation. Deep reinforcement learning models can be trained to deliver the required functionality but are known to suffer from sample inefficiency and slow learning. However, adapting to a partner agent behaviour during the ongoing task requires ability to assess the partner agent type quickly. We suggest a method, where we synthetically produce populations of agents with different behavioural patterns together with ground truth data of their behaviour, and use this data for training a meta-learner. We additionally suggest an agent architecture, which can efficiently use the generated data and gain the meta-learning capability. When an agent is equipped with such a meta-learner, it is capable of quickly adapting to cooperation with unknown partner agent types in new situations. This method can be used to automatically form a task distribution for meta-training from emerging behaviours that arise, for example, through self-play.

References in corpus (12)

• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• A simple neural network module for relational reasoning
• Counterfactual Multi-Agent Policy Gradients
• Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems
• Learning to reinforcement learn
• Emergent Tool Use From Multi-Agent Autocurricula
• A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
• Emergent Complexity via Multi-Agent Competition
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
• "Other-Play" for Zero-Shot Coordination
• Stable Opponent Shaping in Differentiable Games