UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining

Hyung Won Chung, Noah Constant, Xavier Garcia +4

Pretrained multilingual large language models have typically used heuristic temperature-based sampling to balance between different languages. However previous work has not systema…