Unsqueeze [CLS] Bottleneck to Learn Rich Representations

Qing Su, Shihao Ji

Distillation-based self-supervised learning typically leads to more compressed representations due to its radical clustering process and the implementation of a sharper target dist…