Motion-Aware Vision-Reference Alignment for Referring Multi-Object Tracking

Abstract

Referring Multi-Object Tracking (RMOT) extends conventional multi-object tracking (MOT) by introducing natural language references for multi-modal fusion tracking. RMOT benchmarks only describe the object's appearance, relative positions, and initial motion states. This so-called static regulation fails to capture dynamic changes of the object motion, including velocity changes and motion direction shifts. This limitation not only causes a temporal discrepancy between static references and dynamic vision modality but also constrains multi-modal tracking performance. To address this limitation, we propose a novel motion-aware vision-reference alignment framework, named VMRMOT. VMRMOT introduces a motion modality derived from object dynamics to facilitate the alignment between the vision modality and language references. Specifically, we introduce motion-aware descriptions derived from object dynamic behaviors and encode them into motion features as the motion modality through multi-modal large language models (MLLMs). We further design a Vision-Motion-Reference Alignment (VMRA) module to hierarchically align visual queries with motion and reference cues, enhancing their cross-modal consistency. In addition, a Motion-Guided Prediction Head (MGPH) is developed to explore motion modality to enhance the performance of the prediction head. To the best of our knowledge, VMRMOT is the first motion-aware vision-reference alignment framework for the RMOT task. Extensive experiments on multiple RMOT benchmarks demonstrate that VMRMOT outperforms existing state-of-the-art methods. The code is available at https://github.com/Kroery/VMRMOT.

Accepted by IEEE Transactions on Multimedia