3 papers
cs.LG2025
-Parametrization for Mixture of Experts
Jan MaÅaÅnicki, Kamil Ciebiera, Mateusz BoruÅ +8
Recent years have seen a growing interest and adoption of LLMs, with Mixture-of-Experts (MoE) emerging as a leading architecture in extremely large models. Currently, the largest o…
cs.LG2025
Decoupled Relative Learning Rate Schedules
Jan Ludziejewski, Jan MaÅaÅnicki, Maciej Pióro +8
In this work, we introduce a novel approach for optimizing LLM training by adjusting learning rates across weights of different components in Transformer models. Traditional method…
cs.CL2024
Mixture of Tokens: Continuous MoE through Cross-Example Aggregation
Szymon Antoniak, MichaŠKrutul, Maciej Pióro +7
Mixture of Experts (MoE) models based on Transformer architecture are pushing the boundaries of language and vision tasks. The allure of these models lies in their ability to subst…