Agent Modelling under Partial Observability for Deep Reinforcement Learning
arXiv:2006.09447
Abstract
Modelling the behaviours of other agents is essential for understanding how agents interact and making effective decisions. Existing methods for agent modelling commonly assume knowledge of the local observations and chosen actions of the modelled agents during execution. To eliminate this assumption, we extract representations from the local information of the controlled agent using encoder-decoder architectures. Using the observations and actions of the modelled agents during training, our models learn to extract representations about the modelled agents conditioned only on the local observations of the controlled agent. The representations are used to augment the controlled agent's decision policy which is trained via deep reinforcement learning; thus, during execution, the policy does not require access to other agents' information. We provide a comprehensive evaluation and ablations studies in cooperative, competitive and mixed multi-agent environments, showing that our method achieves higher returns than baseline methods which do not use the learned representations.
arXiv commentPublished in the 35th Conference on Neural Information Processing Systems (NeurIPS 2021)
References in corpus (23)
- Auto-Encoding Variational Bayes
- Representation Learning with Contrastive Predictive Coding
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- A Recurrent Latent Variable Model for Sequential Data
- Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
- Recurrent World Models Facilitate Policy Evolution
- Counterfactual Multi-Agent Policy Gradients
- Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems
- Learning to reinforcement learn
- InfoVAE: Information Maximizing Variational Autoencoders
- Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
- Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
- Emergence of Grounded Compositional Language in Multi-Agent Populations
- A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
- Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning
- Learning Plannable Representations with Causal InfoGAN
- Learning Invariant Representations for Reinforcement Learning without Reconstruction
- Learning Policy Representations in Multiagent Systems
- VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
- Relational Forward Models for Multi-Agent Learning
- Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizations
- Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulation
- Decoupling Dynamics and Reward for Transfer Learning