Contrastive Learning with Stronger Augmentations

Abstract

Representation learning has significantly been developed with the advance of contrastive learning methods. Most of those methods have benefited from various data augmentations that are carefully designated to maintain their identities so that the images transformed from the same instance can still be retrieved. However, those carefully designed transformations limited us to further explore the novel patterns exposed by other transformations. Meanwhile, as found in our experiments, the strong augmentations distorted the images' structures, resulting in difficult retrieval. Thus, we propose a general framework called Contrastive Learning with Stronger Augmentations~(CLSA) to complement current contrastive learning approaches. Here, the distribution divergence between the weakly and strongly augmented images over the representation bank is adopted to supervise the retrieval of strongly augmented queries from a pool of instances. Experiments on the ImageNet dataset and downstream datasets showed the information from the strongly augmented images can significantly boost the performance. For example, CLSA achieves top-1 accuracy of 76.2% on ImageNet with a standard ResNet-50 architecture with a single-layer classifier fine-tuned, which is almost the same level as 76.5% of supervised results. The code and pre-trained models are available in https://github.com/maple-research-lab/CLSA.

arXiv comment12 pages, 6 figures

References in corpus (29)

• Auto-Encoding Variational Bayes
• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• mixup: Beyond Empirical Risk Minimization
• Bootstrap your own latent: A new approach to self-supervised Learning
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Unsupervised Representation Learning by Predicting Image Rotations
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Data Augmentation Generative Adversarial Networks
• AutoAugment: Learning Augmentation Policies from Data
• Adversarial Feature Learning
• Learning Representations by Maximizing Mutual Information Across Views
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Prototypical Contrastive Learning of Unsupervised Representations
• Exploring Simple Siamese Representation Learning
• Large Scale Adversarial Representation Learning
• Classification Accuracy Score for Conditional Generative Models
• Self-labelling via simultaneous clustering and representation learning
• EnAET: A Self-Trained framework for Semi-Supervised and Supervised Learning with Ensemble Transformations
• Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination
• AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries
• Spatially Consistent Representation Learning

Cited by in corpus (9)

• Graph Self-Supervised Learning: A Survey
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
• Heterogeneous Contrastive Learning: Encoding Spatial Information for Compact Visual Representations
• An Empirical Study and Analysis on Open-Set Semi-Supervised Learning
• Parametric Contrastive Learning
• Vision Pair Learning: An Efficient Training Framework for Image Classification
• Equivariant Contrastive Learning