On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signals

Haizhou Shi, Youcai Zhang, Siliang Tang +4

It is a consensus that small models perform quite poorly under the paradigm of self-supervised contrastive learning. Existing methods usually adopt a large off-the-shelf model to t…