VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation

Abstract

Behavior prediction in dynamic, multi-agent systems is an important problem in the context of self-driving cars, due to the complex representations and interactions of road components, including moving agents (e.g. pedestrians and vehicles) and road context information (e.g. lanes, traffic lights). This paper introduces VectorNet, a hierarchical graph neural network that first exploits the spatial locality of individual road components represented by vectors and then models the high-order interactions among all components. In contrast to most recent approaches, which render trajectories of moving agents and road context information as bird-eye images and encode them with convolutional neural networks (ConvNets), our approach operates on a vector representation. By operating on the vectorized high definition (HD) maps and agent trajectories, we avoid lossy rendering and computationally intensive ConvNet encoding steps. To further boost VectorNet's capability in learning context features, we propose a novel auxiliary task to recover the randomly masked out map entities and agent trajectories based on their context. We evaluate VectorNet on our in-house behavior prediction benchmark and the recently released Argoverse forecasting dataset. Our method achieves on par or better performance than the competitive rendering approach on both benchmarks while saving over 70% of the model parameters with an order of magnitude reduction in FLOPs. It also outperforms the state of the art on the Argoverse dataset.

CVPR 2020

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• A Recurrent Latent Variable Model for Sequential Data
• IntentNet: Learning to Predict Intention from Raw Sensor Data
• Argoverse: 3D Tracking and Forecasting with Rich Maps
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Multiple Futures Prediction
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision
• Stochastic Prediction of Multi-Agent Interactions from Partial Observations
• VAIN: Attentional Multi-agent Predictive Modeling
• Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions
• Relational Action Forecasting

Cited by in corpus (12)

• A Review of Tracking, Prediction and Decision Making Methods for Autonomous Driving
• THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling
• Learning Lane Graph Representations for Motion Forecasting
• Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies
• What-If Motion Prediction for Autonomous Driving
• Heterogeneous Edge-Enhanced Graph Attention Network For Multi-Agent Trajectory Prediction
• Decoder Fusion RNN: Context and Interaction Aware Decoders for Trajectory Prediction
• PRANK: motion Prediction based on RANKing
• Traffic Forecasting using Vehicle-to-Vehicle Communication
• XInsight: Revealing Model Insights for GNNs with Flow-based Explanations
• Deep Structured Reactive Planning
• Convolutions for Spatial Interaction Modeling