MAST: A Memory-Augmented Self-supervised Tracker

Abstract

Recent interest in self-supervised dense tracking has yielded rapid progress, but performance still remains far from supervised methods. We propose a dense tracking model trained on videos without any annotations that surpasses previous self-supervised methods on existing benchmarks by a significant margin (+15%), and achieves performance comparable to supervised methods. In this paper, we first reassess the traditional choices used for self-supervised training and reconstruction loss by conducting thorough experiments that finally elucidate the optimal choices. Second, we further improve on existing methods by augmenting our architecture with a crucial memory component. Third, we benchmark on large-scale semi-supervised video object segmentation(aka. dense tracking), and propose a new metric: generalizability. Our first two contributions yield a self-supervised network that for the first time is competitive with supervised methods on standard evaluation metrics of dense tracking. When measuring generalizability, we show self-supervised approaches are actually superior to the majority of supervised methods. We believe this new generalizability metric can better capture the real-world use-cases for dense tracking, and will spur new interest in this research direction.

Accepted to CVPR 2020

References in corpus (14)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Pixel Recurrent Neural Networks
• A Novel Performance Evaluation Methodology for Single-Target Trackers
• YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark
• RANet: Ranking Attention Network for Fast Video Object Segmentation
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Get To The Point: Summarization with Pointer-Generator Networks
• Learning visual groups from co-occurrences in space and time
• Learning Video Object Segmentation from Static Images
• Self-supervised Learning for Video Correspondence Flow
• Joint-task Self-supervised Learning for Temporal Correspondence
• Long-term Tracking in the Wild: A Benchmark
• Multigrid Predictive Filter Flow for Unsupervised Learning on Videos

Cited by in corpus (4)

• Space-Time Correspondence as a Contrastive Random Walk
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• 3D-OES: Viewpoint-Invariant Object-Factorized Environment Simulators
• Self-supervised Video Object Segmentation