Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

Atindra Jha, Margaret Li, Jure Leskovec +2

As the supply of human-written text is exhausted, it has become standard practice to repeat language model training data. Prior work has studied data repetition for densely activat…