A Closer Look at Evaluation of Text-to-Video Models Under Camera Motion

Abstract

Existing text-to-video (T2V) evaluation benchmarks, such as VBench and EvalCrafter, predominantly contain either subject-centric prompts or static camera scenes, overlooking the importance of camera motion, essential for producing cinematic shots with realistic scenes. Due to this, the metrics developed in these works often produce incorrect results in the presence of camera motion. To fill this gap, we introduce DynamicEval, a benchmark comprising of systematically curated prompts that emphasize dynamic camera motion, paired with 45k human annotations on video pairs. DynamicEval evaluates two key dimensions of video quality: the consistency of the foreground subjects and the consistency of the background scene. To address the shortcomings of existing metrics under camera motion, we introduce two novel metrics: 1) Tracker-FG, which evaluates subject consistency by assessing the keypoint tracks of the subject. 2) Debiased Motion Smoothness, for background consistency, devoid from issues of occlusions/disocclusions arising from camera and foreground subject movements. Extensive experiments on the large-scale DynamicEval benchmark demonstrate that our proposed metrics achieve stronger correlations with human preferences at both the video level, and the model level with an increase of 0.32 in average Pearson Linear Correlation.

arXiv commentAccepted at WACV 2027. 22 pages, 15 figures, 13 tables. Access the project page in https://nithincbabu7.github.io/DynamicEval