BYOL works even without batch statistics

Abstract

Bootstrap Your Own Latent (BYOL) is a self-supervised learning approach for image representation. From an augmented view of an image, BYOL trains an online network to predict a target network representation of a different augmented view of the same image. Unlike contrastive methods, BYOL does not explicitly use a repulsion term built from negative pairs in its training objective. Yet, it avoids collapse to a trivial, constant representation. Thus, it has recently been hypothesized that batch normalization (BN) is critical to prevent collapse in BYOL. Indeed, BN flows gradients across batch elements, and could leak information about negative views in the batch, which could act as an implicit negative (contrastive) term. However, we experimentally show that replacing BN with a batch-independent normalization scheme (namely, a combination of group normalization and weight standardization) achieves performance comparable to vanilla BYOL ($73.9\%$ vs. $74.3\%$ top-1 accuracy under the linear evaluation protocol on ImageNet with ResNet-$50$). Our finding disproves the hypothesis that the use of batch statistics is a crucial ingredient for BYOL to learn useful representations.

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Large Batch Training of Convolutional Networks
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Data-dependent Initializations of Convolutional Neural Networks
• On Variational Bounds of Mutual Information
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Understanding Self-supervised Learning with Dual Deep Networks

Cited by in corpus (11)

• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• Whitening for Self-Supervised Representation Learning
• Understanding Self-supervised Learning with Dual Deep Networks
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• Self-supervised Graph Neural Networks without explicit negative sampling
• EqCo: Equivalent Rules for Self-supervised Contrastive Learning
• On Feature Decorrelation in Self-Supervised Learning
• An Empirical Study of Graph Contrastive Learning
• Object-Aware Cropping for Self-Supervised Learning