2 papers
cs.CL2026
Compute Optimal Tokenization
Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer +6
Scaling laws enable the optimal selection of data amount and language model size, yet the impact of the data unit, the token, on this relationship remains underexplored. In this wo…
cs.LG2026
Improving MoE Compute Efficiency by Composing Weight and Data Sparsity
Maciej Kilian, Oleg Mkrtchyan, Luke Zettlemoyer +2
Mixture-of-Experts layers achieve compute efficiency through weight sparsity: each token activates only a subset of experts. Data sparsity, where each expert processes only a subse…