Domain-Agnostic Clustering with Self-Distillation

Abstract

Recent advancements in self-supervised learning have reduced the gap between supervised and unsupervised representation learning. However, most self-supervised and deep clustering techniques rely heavily on data augmentation, rendering them ineffective for many learning tasks where insufficient domain knowledge exists for performing augmentation. We propose a new self-distillation based algorithm for domain-agnostic clustering. Our method builds upon the existing deep clustering frameworks and requires no separate student model. The proposed method outperforms existing domain agnostic (augmentation-free) algorithms on CIFAR-10. We empirically demonstrate that knowledge distillation can improve unsupervised representation learning by extracting richer `dark knowledge' from the model than using predicted labels alone. Preliminary experiments also suggest that self-distillation improves the convergence of DeepCluster-v2.

arXiv commentNeurIPS 2021 Workshop: Self-Supervised Learning - Theory and Practice

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• mixup: Beyond Empirical Risk Minimization
• Deep Residual Learning for Image Recognition
• Knowledge Distillation: A Survey
• Bootstrap your own latent: A new approach to self-supervised Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Data Augmentation using Random Image Cropping and Patching for Deep CNNs
• Deep Clustering for Unsupervised Learning of Visual Features
• Data augmentation instead of explicit regularization
• Learning Lightweight Lane Detection CNNs by Self Attention Distillation
• Viewmaker Networks: Learning Views for Unsupervised Representation Learning
• Snapshot Distillation: Teacher-Student Optimization in One Generation
• DABS: A Domain-Agnostic Benchmark for Self-Supervised Learning
• ClusterFit: Improving Generalization of Visual Representations