OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning

Abstract

Learning image representations without human supervision is an important and active research field. Several recent approaches have successfully leveraged the idea of making such a representation invariant under different types of perturbations, especially via contrastive-based instance discrimination training. Although effective visual representations should indeed exhibit such invariances, there are other important characteristics, such as encoding contextual reasoning skills, for which alternative reconstruction-based approaches might be better suited. With this in mind, we propose a teacher-student scheme to learn representations by training a convolutional net to reconstruct a bag-of-visual-words (BoW) representation of an image, given as input a perturbed version of that same image. Our strategy performs an online training of both the teacher network (whose role is to generate the BoW targets) and the student network (whose role is to learn representations), along with an online update of the visual-words vocabulary (used for the BoW targets). This idea effectively enables fully online BoW-guided unsupervised learning. Extensive experiments demonstrate the interest of our BoW-based strategy which surpasses previous state-of-the-art methods (including contrastive-based ones) in several applications. For instance, in downstream tasks such Pascal object detection, Pascal classification and Places205 classification, our method improves over all prior unsupervised approaches, thus establishing new state-of-the-art results that are also significantly better even than those of supervised pre-training. We provide the implementation code at https://github.com/valeoai/obow.

arXiv commentAccepted to CVPR2021. Code at https://github.com/valeoai/obow

References in corpus (15)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• Prototypical Networks for Few-shot Learning
• Neural Discrete Representation Learning
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Interpolation Consistency Training for Semi-Supervised Learning
• What Makes for Good Views for Contrastive Learning?
• Large Scale Adversarial Representation Learning
• Bayesian Dark Knowledge
• Selfie: Self-supervised Pretraining for Image Embedding
• A Framework For Contrastive Self-Supervised Learning And Designing A New Approach
• Unsupervised Learning of Dense Visual Representations
• Distilling Model Knowledge