Growing to See: Efficient Pretraining of Self-Supervised Vision Transformers

Amir Mohammad Karimi Mamaghan, Ferdinand Kapl, Andrea Dittadi +4

Progressive depth growth builds large networks by starting with shallow models and inserting layers during training. Recent advances in large language models have renewed interest…