Tracking Without Re-recognition in Humans and Machines

Abstract

Imagine trying to track one particular fruitfly in a swarm of hundreds. Higher biological visual systems have evolved to track moving objects by relying on both appearance and motion features. We investigate if state-of-the-art deep neural networks for visual tracking are capable of the same. For this, we introduce PathTracker, a synthetic visual challenge that asks human observers and machines to track a target object in the midst of identical-looking "distractor" objects. While humans effortlessly learn PathTracker and generalize to systematic variations in task design, state-of-the-art deep networks struggle. To address this limitation, we identify and model circuit mechanisms in biological brains that are implicated in tracking objects based on motion cues. When instantiated as a recurrent network, our circuit model learns to solve PathTracker with a robust visual strategy that rivals human performance and explains a significant proportion of their decision-making on the challenge. We also show that the success of this circuit model extends to object tracking in natural videos. Adding it to a transformer-based architecture for object tracking builds tolerance to visual nuisances that affect object appearance, resulting in a new state-of-the-art performance on the large-scale TrackingNet object tracking challenge. Our work highlights the importance of building artificial vision models that can help us better understand human vision and improve computer vision.

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Deep Residual Learning for Image Recognition
• Shortcut Learning in Deep Neural Networks
• GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild
• Is Space-Time Attention All You Need for Video Understanding?
• MOT16: A Benchmark for Multi-Object Tracking
• A Closer Look at Spatiotemporal Convolutions for Action Recognition
• Long Range Arena: A Benchmark for Efficient Transformers
• Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking
• Tracking Noisy Targets: A Review of Recent Object Tracking Approaches
• Hierarchical Attentive Recurrent Tracking
• Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency
• LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking
• Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning
• A Connectome Based Hexagonal Lattice Convolutional Network Model of the Drosophila Visual System
• Recurrent neural circuits for contour detection