Generating Multi-Agent Trajectories using Programmatic Weak Supervision

Abstract

We study the problem of training sequential generative models for capturing coordinated multi-agent trajectory behavior, such as offensive basketball gameplay. When modeling such settings, it is often beneficial to design hierarchical models that can capture long-term coordination using intermediate variables. Furthermore, these intermediate variables should capture interesting high-level behavioral semantics in an interpretable and manipulatable way. We present a hierarchical framework that can effectively learn such sequential generative models. Our approach is inspired by recent work on leveraging programmatically produced weak labels, which we extend to the spatiotemporal regime. In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods. We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts.

References in corpus (23)

• Auto-Encoding Variational Bayes
• WaveNet: A Generative Model for Raw Audio
• Categorical Reparameterization with Gumbel-Softmax
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Pixel Recurrent Neural Networks
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Snorkel: Rapid Training Data Creation with Weak Supervision
• A note on the evaluation of generative models
• Data Programming: Creating Large Training Sets, Quickly
• Generative Adversarial Imitation Learning
• Variational Lossy Autoencoder
• Edward: A library for probabilistic modeling, inference, and criticism
• Composing graphical models with neural networks for structured representations and fast inference
• A Hierarchical Neural Autoencoder for Paragraphs and Documents
• Sequential Neural Models with Stochastic Layers
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations
• Learning Features of Music from Scratch
• Learning the Structure of Generative Models without Labeled Data
• Strategic Attentive Writer for Learning Macro-Actions
• Factorized Point Process Intensities: A Spatial Analysis of Professional Basketball
• Learning recurrent representations for hierarchical behavior modeling

Cited by in corpus (20)

• A Review on Deep Learning Techniques for Video Prediction
• TNT: Target-driveN Trajectory Prediction
• VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation
• AC-VRNN: Attentive Conditional-VRNN for Multi-Future Trajectory Prediction
• Data-driven Analysis for Understanding Team Sports Behaviors
• Unsupervised Learning of Object Structure and Dynamics from Videos
• Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods
• NAOMI: Non-Autoregressive Multiresolution Sequence Imputation
• RetinaTrack: Online Single Stage Joint Detection and Tracking
• On Multi-Agent Learning in Team Sports Games
• Ball Trajectory Inference from Multi-Agent Sports Contexts Using Set Transformer and Hierarchical Bi-LSTM
• Decentralized policy learning with partial observation and mechanical constraints for multiperson modeling
• Imitation Learning: Progress, Taxonomies and Challenges
• baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotemporal Modeling
• About latent roles in forecasting players in team sports
• baller2vec++: A Look-Ahead Multi-Entity Transformer For Modeling Coordinated Agents
• Creating Training Sets via Weak Indirect Supervision
• Learning Calibratable Policies using Programmatic Style-Consistency
• Imitation Learning of Factored Multi-agent Reactive Models
• Multi-Agent Imitation Learning with Copulas