ClusterFit: Improving Generalization of Visual Representations
arXiv:1912.03330
Abstract
Pre-training convolutional neural networks with weakly-supervised and self-supervised strategies is becoming increasingly popular for several computer vision tasks. However, due to the lack of strong discriminative signals, these learned representations may overfit to the pre-training objective (e.g., hashtag prediction) and not generalize well to downstream tasks. In this work, we present a simple strategy - ClusterFit (CF) to improve the robustness of the visual representations learned during pre-training. Given a dataset, we (a) cluster its features extracted from a pre-trained network using k-means and (b) re-train a new network from scratch on this dataset using cluster assignments as pseudo-labels. We empirically show that clustering helps reduce the pre-training task-specific information from the extracted features thereby minimizing overfitting to the same. Our approach is extensible to different pre-training frameworks -- weak- and self-supervised, modalities -- images and videos, and pre-training tasks -- object and action classification. Through extensive transfer learning experiments on 11 different target datasets of varied vocabularies and granularities, we show that ClusterFit significantly improves the representation quality compared to the state-of-the-art large-scale (millions / billions) weakly-supervised image and video models and self-supervised image models.
References in corpus (12)
- Distilling the Knowledge in a Neural Network
- Prototypical Networks for Few-shot Learning
- The Kinetics Human Action Video Dataset
- YFCC100M: The New Data in Multimedia Research
- Do Deep Nets Really Need to be Deep?
- Born Again Neural Networks
- Billion-scale semi-supervised learning for image classification
- Large scale distributed neural network training through online distillation
- Revisiting Self-Supervised Visual Representation Learning
- Colorization as a Proxy Task for Visual Understanding
- Predicting Deeper into the Future of Semantic Segmentation
- Hard Mixtures of Experts for Large Scale Weakly Supervised Vision