Attention-Only White-Box Transformer via LeJEPA-Based Self-Supervised Pretraining

Yang Bai, Linyuan Wang, Haoyang Jiang +3

Existing studies on self-supervised learning for white-box networks typically decouple the derivation of white-box networks via optimization algorithms from self-supervised learnin…