Dynamic Relational Inference in Multi-Agent Trajectories

Abstract

Inferring interactions from multi-agent trajectories has broad applications in physics, vision and robotics. Neural relational inference (NRI) is a deep generative model that can reason about relations in complex dynamics without supervision. In this paper, we take a careful look at this approach for relational inference in multi-agent trajectories. First, we discover that NRI can be fundamentally limited without sufficient long-term observations. Its ability to accurately infer interactions degrades drastically for short output sequences. Next, we consider a more general setting of relational inference when interactions are changing overtime. We propose an extension ofNRI, which we call the DYnamic multi-AgentRelational Inference (DYARI) model that can reason about dynamic relations. We conduct exhaustive experiments to study the effect of model architecture, under-lying dynamics and training scheme on the performance of dynamic relational inference using a simulated physics system. We also showcase the usage of our model on real-world multi-agent basketball trajectories.

arXiv commentsubmitted to ICLR 2021

References in corpus (18)

• Auto-Encoding Variational Bayes
• Graph Embedding Techniques, Applications, and Performance: A Survey
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
• A Recurrent Latent Variable Model for Sequential Data
• Interaction Networks for Learning about Objects, Relations and Physics
• Learning to Simulate Complex Physics with Graph Networks
• Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems
• Causal Effect Inference with Deep Latent-Variable Models
• Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
• Flexible Neural Representation for Physics Prediction
• Multi-Agent Generative Adversarial Imitation Learning
• Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data
• Learning Policy Representations in Multiagent Systems
• Coordinated Multi-Agent Imitation Learning
• Relational Forward Models for Multi-Agent Learning
• Neural Relational Inference with Fast Modular Meta-learning
• Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules