6 papers
Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
Jakub Krajewski, Amitis Shidani, Dan Busbridge +2
While scaling laws for Large Language Models (LLMs) traditionally focus on proxy metrics like pretraining loss, predicting downstream task performance has been considered unreliabl…
-Parametrization for Mixture of Experts
Jan MaÅaÅnicki, Kamil Ciebiera, Mateusz BoruÅ +8
Recent years have seen a growing interest and adoption of LLMs, with Mixture-of-Experts (MoE) emerging as a leading architecture in extremely large models. Currently, the largest o…
Decoupled Relative Learning Rate Schedules
Jan Ludziejewski, Jan MaÅaÅnicki, Maciej Pióro +8
In this work, we introduce a novel approach for optimizing LLM training by adjusting learning rates across weights of different components in Transformer models. Traditional method…
Projected Compression: Trainable Projection for Efficient Transformer Compression
Maciej Stefaniak, MichaÅ Krutul, Jan MaÅaÅnicki +6
Large language models have steadily increased in size to achieve improved performance; however, this growth has also led to greater inference time and computational demands. Conseq…
Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
Jakub Krajewski, Marcin Chochowski, Daniel Korzekwa
Mixture of Experts (MoE) architectures have emerged as pivotal for scaling Large Language Models (LLMs) efficiently. Fine-grained MoE approaches - utilizing more numerous, smaller…
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
Jan Ludziejewski, Maciej Pióro, Jakub Krajewski +8
Mixture of Experts (MoE) architectures have significantly increased computational efficiency in both research and real-world applications of large-scale machine learning models. Ho…