Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations

Grégoire Mialon, Randall Balestriero, Yann LeCun

Self-Supervised Learning (SSL) methods such as VICReg, Barlow Twins or W-MSE avoid collapse of their joint embedding architectures by constraining or regularizing the covariance ma…