Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data

Abstract

This paper studies the problem of novel category discovery on single- and multi-modal data with labels from different but relevant categories. We present a generic, end-to-end framework to jointly learn a reliable representation and assign clusters to unlabelled data. To avoid over-fitting the learnt embedding to labelled data, we take inspiration from self-supervised representation learning by noise-contrastive estimation and extend it to jointly handle labelled and unlabelled data. In particular, we propose using category discrimination on labelled data and cross-modal discrimination on multi-modal data to augment instance discrimination used in conventional contrastive learning approaches. We further employ Winner-Take-All (WTA) hashing algorithm on the shared representation space to generate pairwise pseudo labels for unlabelled data to better predict cluster assignments. We thoroughly evaluate our framework on large-scale multi-modal video benchmarks Kinetics-400 and VGG-Sound, and image benchmarks CIFAR10, CIFAR100 and ImageNet, obtaining state-of-the-art results.

arXiv commentICCV 2021

References in corpus (13)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• The Kinetics Human Action Video Dataset
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Contrastive Multiview Coding
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Self-Supervised MultiModal Versatile Networks
• AutoNovel: Automatically Discovering and Learning Novel Visual Categories
• Learning to cluster in order to transfer across domains and tasks
• Supervised Contrastive Learning
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation
• Automatically Discovering and Learning New Visual Categories with Ranking Statistics
• A probabilistic constrained clustering for transfer learning and image category discovery