GenURL: A General Framework for Unsupervised Representation Learning

Abstract

Unsupervised representation learning (URL), which learns compact embeddings of high-dimensional data without supervision, has made remarkable progress recently. However, the development of URLs for different requirements is independent, which limits the generalization of the algorithms, especially prohibitive as the number of tasks grows. For example, dimension reduction methods, t-SNE, and UMAP optimize pair-wise data relationships by preserving the global geometric structure, while self-supervised learning, SimCLR, and BYOL focus on mining the local statistics of instances under specific augmentations. To address this dilemma, we summarize and propose a unified similarity-based URL framework, GenURL, which can smoothly adapt to various URL tasks. In this paper, we regard URL tasks as different implicit constraints on the data geometric structure that help to seek optimal low-dimensional representations that boil down to data structural modeling (DSM) and low-dimensional transformation (LDT). Specifically, DMS provides a structure-based submodule to describe the global structures, and LDT learns compact low-dimensional embeddings with given pretext tasks. Moreover, an objective function, General Kullback-Leibler divergence (GKL), is proposed to connect DMS and LDT naturally. Comprehensive experiments demonstrate that GenURL achieves consistent state-of-the-art performance in self-supervised visual learning, unsupervised knowledge distillation (KD), graph embeddings (GE), and dimension reduction.

TNNLS 2024 version with 13 pages and 14 figures

References in corpus (17)

• Distilling the Knowledge in a Neural Network
• Semi-Supervised Classification with Graph Convolutional Networks
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• A Simple Framework for Contrastive Learning of Visual Representations
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Bootstrap your own latent: A new approach to self-supervised Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Learning Graph Embedding with Adversarial Training Methods
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Graph InfoClust: Leveraging cluster-level node information for unsupervised graph representation learning
• Deep Mutual Learning
• Attributed Graph Clustering via Adaptive Graph Convolution
• Extendable and invertible manifold learning with geometry regularized autoencoders