Rethinking Trajectory Forecasting Evaluation

Abstract

Forecasting the behavior of other agents is an integral part of the modern robotic autonomy stack, especially in safety-critical scenarios with human-robot interaction, such as autonomous driving. In turn, there has been a significant amount of interest and research in trajectory forecasting, resulting in a wide variety of approaches. Common to all works, however, is the use of the same few accuracy-based evaluation metrics, e.g., displacement error and log-likelihood. While these metrics are informative, they are task-agnostic and predictions that are evaluated as equal can lead to vastly different outcomes, e.g., in downstream planning and decision making. In this work, we take a step back and critically evaluate current trajectory forecasting metrics, proposing task-aware metrics as a better measure of performance in systems where prediction is being deployed. We additionally present one example of such a metric, incorporating planning-awareness within existing trajectory forecasting metrics.

arXiv comment4 pages, 2 figures

References in corpus (6)

• Towards A Rigorous Science of Interpretable Machine Learning
• Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net
• A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving
• Collision Avoidance in Pedestrian-Rich Environments with Deep Reinforcement Learning
• Continuous Inverse Optimal Control with Locally Optimal Examples
• Survey on Vision-based Path Prediction