Self-Distillation for Further Pre-training of Transformers

Seanie Lee, Minki Kang, Juho Lee +2

Pre-training a large transformer model on a massive amount of unlabeled data and fine-tuning it on labeled datasets for diverse downstream tasks has proven to be a successful strat…