Cross-Lingual Supervision improves Large Language Models Pre-training

Andrea Schioppa, Xavier Garcia, Orhan Firat

The recent rapid progress in pre-training Large Language Models has relied on using self-supervised language modeling objectives like next token prediction or span corruption. On t…