Temporal Knowledge Consistency for Unsupervised Visual Representation Learning

Abstract

The instance discrimination paradigm has become dominant in unsupervised learning. It always adopts a teacher-student framework, in which the teacher provides embedded knowledge as a supervision signal for the student. The student learns meaningful representations by enforcing instance spatial consistency with the views from the teacher. However, the outputs of the teacher can vary dramatically on the same instance during different training stages, introducing unexpected noise and leading to catastrophic forgetting caused by inconsistent objectives. In this paper, we first integrate instance temporal consistency into current instance discrimination paradigms, and propose a novel and strong algorithm named Temporal Knowledge Consistency (TKC). Specifically, our TKC dynamically ensembles the knowledge of temporal teachers and adaptively selects useful information according to its importance to learning instance temporal consistency. Experimental result shows that TKC can learn better visual representations on both ResNet and AlexNet on linear evaluation protocol while transfer well to downstream tasks. All experiments suggest the good effectiveness and generalization of our method.

arXiv commentTo appear in ICCV 2021

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Unsupervised Representation Learning by Predicting Image Rotations
• Temporal Ensembling for Semi-Supervised Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Multiview Coding
• Exploring Simple Siamese Representation Learning
• Large Scale Adversarial Representation Learning
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Hard Negative Mixing for Contrastive Learning
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• Delving into Inter-Image Invariance for Unsupervised Visual Representations
• Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning