Imitation Learning of Factored Multi-agent Reactive Models
arXiv:1903.04714
Abstract
We apply recent advances in deep generative modeling to the task of imitation learning from biological agents. Specifically, we apply variations of the variational recurrent neural network model to a multi-agent setting where we learn policies of individual uncoordinated agents acting based on their perceptual inputs and their hidden belief state. We learn stochastic policies for these agents directly from observational data, without constructing a reward function. An inference network learned jointly with the policy allows for efficient inference over the agent's belief state given a sequence of its current perceptual inputs and the prior actions it performed, which lets us extrapolate observed sequences of behavior into the future while maintaining uncertainty estimates over future trajectories. We test our approach on a dataset of flies interacting in a 2D environment, where we demonstrate better predictive performance than existing approaches which learn deterministic policies with recurrent neural networks. We further show that the uncertainty estimates over future trajectories we obtain are well calibrated, which makes them useful for a variety of downstream processing tasks.
arXiv commentincorporated into another paper with different motivations
References in corpus (13)
- Auto-Encoding Variational Bayes
- Categorical Reparameterization with Gumbel-Softmax
- Variational Inference with Normalizing Flows
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- A Recurrent Latent Variable Model for Sequential Data
- Emergence of grid-like representations by training recurrent neural networks to perform spatial localization
- REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
- Variational inference for Monte Carlo objectives
- Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
- Generating Multi-Agent Trajectories using Programmatic Weak Supervision
- Generative Models of Visually Grounded Imagination
- Generative Temporal Models with Memory
- Revisiting Reweighted Wake-Sleep for Models with Stochastic Control Flow