Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation

Joel Dag, Nikit Srivastava, Hamada M. Zahera +1

Multilingual models share capacity across languages, so adding languages can hurt performance on existing ones. Mixture-of-experts models expand this capacity with little additiona…