baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotemporal Modeling

Abstract

Multi-agent spatiotemporal modeling is a challenging task from both an algorithmic design and computational complexity perspective. Recent work has explored the efficacy of traditional deep sequential models in this domain, but these architectures are slow and cumbersome to train, particularly as model size increases. Further, prior attempts to model interactions between agents across time have limitations, such as imposing an order on the agents, or making assumptions about their relationships. In this paper, we introduce baller2vec, a multi-entity generalization of the standard Transformer that can, with minimal assumptions, simultaneously and efficiently integrate information across entities and time. We test the effectiveness of baller2vec for multi-agent spatiotemporal modeling by training it to perform two different basketball-related tasks: (1) simultaneously modeling the trajectories of all players on the court and (2) modeling the trajectory of the ball. Not only does baller2vec learn to perform these tasks well (outperforming a graph recurrent neural network with a similar number of parameters by a wide margin), it also appears to "understand" the game of basketball, encoding idiosyncratic qualities of players in its embeddings, and performing basketball-relevant functions with its attention heads.

References in corpus (13)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Distributed Representations of Sentences and Documents
• Language Models are Few-Shot Learners
• A Recurrent Latent Variable Model for Sequential Data
• Layer Normalization
• Language Modeling with Deep Transformers
• Generating Long-term Trajectories Using Deep Hierarchical Networks
• Coordinated Multi-Agent Imitation Learning
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision
• baller2vec++: A Look-Ahead Multi-Entity Transformer For Modeling Coordinated Agents

Cited by in corpus (5)

• Keyword Transformer: A Self-Attention Model for Keyword Spotting
• Data-driven Analysis for Understanding Team Sports Behaviors
• baller2vec++: A Look-Ahead Multi-Entity Transformer For Modeling Coordinated Agents
• Time-series Imputation of Temporally-occluded Multiagent Trajectories
• The DEformer: An Order-Agnostic Distribution Estimating Transformer