Compressive Visual Representations

Abstract

Learning effective visual representations that generalize well without human supervision is a fundamental problem in order to apply Machine Learning to a wide variety of tasks. Recently, two families of self-supervised methods, contrastive learning and latent bootstrapping, exemplified by SimCLR and BYOL respectively, have made significant progress. In this work, we hypothesize that adding explicit information compression to these algorithms yields better and more robust representations. We verify this by developing SimCLR and BYOL formulations compatible with the Conditional Entropy Bottleneck (CEB) objective, allowing us to both measure and control the amount of compression in the learned representation, and observe their impact on downstream tasks. Furthermore, we explore the relationship between Lipschitz continuity and compression, showing a tractable lower bound on the Lipschitz constant of the encoders we learn. As Lipschitz continuity is closely related to robustness, this provides a new explanation for why compressed models are more robust. Our experiments confirm that adding compression to SimCLR and BYOL significantly improves linear evaluation accuracies and model robustness across a wide range of domain shifts. In particular, the compressed version of BYOL achieves 76.0% Top-1 linear evaluation accuracy on ImageNet with ResNet-50, and 78.8% with ResNet-50 2x.

arXiv commentNeurIPS 2021. 27 pages, 4 figures. Code and pretrained models at https://github.com/google-research/compressive-visual-representations

References in corpus (21)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• The information bottleneck method
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Wide Residual Networks
• Learning deep representations by mutual information estimation and maximization
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Fine-Grained Visual Classification of Aircraft
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Large Batch Training of Convolutional Networks
• Do ImageNet Classifiers Generalize to ImageNet?
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• On Variational Bounds of Mutual Information
• The Conditional Entropy Bottleneck
• von Mises-Fisher Mixture Model-based Deep learning: Application to Face Verification
• Learning Optimal Representations with the Decodable Information Bottleneck
• The Role of Information Complexity and Randomization in Representation Learning
• Lossy Compression for Lossless Prediction