Supporting Clustering with Contrastive Learning
arXiv:2103.12953
Abstract
Unsupervised clustering aims at discovering the semantic categories of data according to some distance measured in the representation space. However, different categories often overlap with each other in the representation space at the beginning of the learning process, which poses a significant challenge for distance-based clustering in achieving good separation between different categories. To this end, we propose Supporting Clustering with Contrastive Learning (SCCL) -- a novel framework to leverage contrastive learning to promote better separation. We assess the performance of SCCL on short text clustering and show that SCCL significantly advances the state-of-the-art results on most benchmark datasets with 3%-11% improvement on Accuracy and 4%-15% improvement on Normalized Mutual Information. Furthermore, our quantitative analysis demonstrates the effectiveness of SCCL in leveraging the strengths of both bottom-up instance discrimination and top-down clustering to achieve better intra-cluster and inter-cluster distances when evaluated with the ground truth cluster labels.
arXiv commentNAACL 2021
References in corpus (20)
- Efficient Estimation of Word Representations in Vector Space
- Distributed Representations of Words and Phrases and their Compositionality
- A Simple Framework for Contrastive Learning of Visual Representations
- mixup: Beyond Empirical Risk Minimization
- Improved Baselines with Momentum Contrastive Learning
- Prototypical Contrastive Learning of Unsupervised Representations
- Text Understanding from Scratch
- Hard Negative Mixing for Contrastive Learning
- Self-Taught Convolutional Neural Networks for Short Text Clustering
- Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
- COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
- Supervised Contrastive Learning
- SpectralNet: Spectral Clustering using Deep Neural Networks
- DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering
- Mixture Models for Diverse Machine Translation: Tricks of the Trade
- i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning
- Deep Unsupervised Clustering Using Mixture of Autoencoders
- CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding