How To Train Your Deep Multi-Object Tracker

Abstract

The recent trend in vision-based multi-object tracking (MOT) is heading towards leveraging the representational power of deep learning to jointly learn to detect and track objects. However, existing methods train only certain sub-modules using loss functions that often do not correlate with established tracking evaluation measures such as Multi-Object Tracking Accuracy (MOTA) and Precision (MOTP). As these measures are not differentiable, the choice of appropriate loss functions for end-to-end training of multi-object tracking methods is still an open research problem. In this paper, we bridge this gap by proposing a differentiable proxy of MOTA and MOTP, which we combine in a loss function suitable for end-to-end training of deep multi-object trackers. As a key ingredient, we propose a Deep Hungarian Net (DHN) module that approximates the Hungarian matching algorithm. DHN allows estimating the correspondence between object tracks and ground truth objects to compute differentiable proxies of MOTA and MOTP, which are in turn used to optimize deep trackers directly. We experimentally demonstrate that the proposed differentiable framework improves the performance of existing multi-object trackers, and we establish a new state of the art on the MOTChallenge benchmark. Our code is publicly available from https://github.com/yihongXU/deepMOT.

arXiv comment14 pages, 9 figures, 6 tables

References in corpus (10)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• MOT16: A Benchmark for Multi-Object Tracking
• MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking
• SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks
• Distractor-aware Siamese Networks for Visual Object Tracking
• An On-line Variational Bayesian Model for Multi-Person Tracking from Cluttered Scenes
• Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor
• Trajectory Factory: Tracklet Cleaving and Re-connection by Deep Siamese Bi-GRU for Multiple Object Tracking
• Fusion of Head and Full-Body Detectors for Multi-Object Tracking

Cited by in corpus (10)

• 3D Multi-Object Tracking: A Baseline and New Evaluation Metrics
• Multi-Object Tracking with Siamese Track-RCNN
• SMOT: Single-Shot Multi Object Tracking
• ArTIST: Autoregressive Trajectory Inpainting and Scoring for Tracking
• SiamMOT: Siamese Multi-Object Tracking
• SAMOT: Switcher-Aware Multi-Object Tracking and Still Another MOT Measure
• Multiple Object Tracking with Motion and Appearance Cues
• EagerMOT: 3D Multi-Object Tracking via Sensor Fusion
• Map3D: Registration Based Multi-Object Tracking on 3D Serial Whole Slide Images
• A Hypergradient Approach to Robust Regression without Correspondence