A Framework For Contrastive Self-Supervised Learning And Designing A New Approach

Abstract

Contrastive self-supervised learning (CSL) is an approach to learn useful representations by solving a pretext task that selects and compares anchor, negative and positive (APN) features from an unlabeled dataset. We present a conceptual framework that characterizes CSL approaches in five aspects (1) data augmentation pipeline, (2) encoder selection, (3) representation extraction, (4) similarity measure, and (5) loss function. We analyze three leading CSL approaches--AMDIM, CPC, and SimCLR--, and show that despite different motivations, they are special cases under this framework. We show the utility of our framework by designing Yet Another DIM (YADIM) which achieves competitive results on CIFAR-10, STL-10 and ImageNet, and is more robust to the choice of encoder and the representation extraction strategy. To support ongoing CSL research, we release the PyTorch implementation of this conceptual framework along with standardized implementations of AMDIM, CPC (V2), SimCLR, BYOL, Moco (V2) and YADIM.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Baselines with Momentum Contrastive Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Learning Representations by Maximizing Mutual Information Across Views
• Contrastive Multiview Coding
• Large Batch Training of Convolutional Networks
• Learning to See by Moving
• Learning visual groups from co-occurrences in space and time
• Revisiting Self-Supervised Visual Representation Learning

Cited by in corpus (19)

• Graph Contrastive Learning with Adaptive Augmentation
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• Hard Negative Mixing for Contrastive Learning
• Self-supervised representation learning from 12-lead ECG data
• VoxSRC 2020: The Second VoxCeleb Speaker Recognition Challenge
• Understanding Self-supervised Learning with Dual Deep Networks
• In-Distribution and Out-of-Distribution Self-supervised ECG Representation Learning for Arrhythmia Detection
• EM-driven unsupervised learning for efficient motion segmentation
• CoLES: Contrastive Learning for Event Sequences with Self-Supervision
• The DKU-DukeECE Systems for VoxCeleb Speaker Recognition Challenge 2020
• enpheeph: A Fault Injection Framework for Spiking and Compressed Deep Neural Networks
• The Causal-Neural Connection: Expressiveness, Learnability, and Inference
• The DKU-DukeECE System for the Self-Supervision Speaker Verification Task of the 2021 VoxCeleb Speaker Recognition Challenge
• SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning
• For self-supervised learning, Rationality implies generalization, provably
• An iterative framework for self-supervised deep speaker representation learning
• Sensor Data Augmentation by Resampling for Contrastive Learning in Human Activity Recognition
• Improving Compositionality of Neural Networks by Decoding Representations to Inputs
• OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning