Multi-Agent Reinforcement Learning with Multi-Step Generative Models
arXiv:1901.10251
Abstract
We consider model-based reinforcement learning (MBRL) in 2-agent, high-fidelity continuous control problems -- an important domain for robots interacting with other agents in the same workspace. For non-trivial dynamical systems, MBRL typically suffers from accumulating errors. Several recent studies have addressed this problem by learning latent variable models for trajectory segments and optimizing over behavior in the latent space. In this work, we investigate whether this approach can be extended to 2-agent competitive and cooperative settings. The fundamental challenge is how to learn models that capture interactions between agents, yet are disentangled to allow for optimization of each agent behavior separately. We propose such models based on a disentangled variational auto-encoder, and demonstrate our approach on a simulated 2-robot manipulation task, where one robot can either help or distract the other. We show that our approach has better sample efficiency than a strong model-free RL baseline, and can learn both cooperative and adversarial behavior from the same data.
References in corpus (11)
- Auto-Encoding Variational Bayes
- WaveNet: A Generative Model for Raw Audio
- InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
- Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
- Unity: A General Platform for Intelligent Agents
- Model-Ensemble Trust-Region Policy Optimization
- Emergence of Grounded Compositional Language in Multi-Agent Populations
- DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents
- Learning Latent Subspaces in Variational Autoencoders
- Deep Imitative Models for Flexible Inference, Planning, and Control
- Socially Aware Motion Planning with Deep Reinforcement Learning