2 citations · 2 across the 4 of their papers we have counts for
5 papers
-Parametrization for Mixture of Experts
Jan Małaśnicki, Kamil Ciebiera, Mateusz Boruń +8
Recent years have seen a growing interest and adoption of LLMs, with Mixture-of-Experts (MoE) emerging as a leading architecture in extremely large models. Currently, the largest o…
Decoupled Relative Learning Rate Schedules
Jan Ludziejewski, Jan Małaśnicki, Maciej Pióro +8
In this work, we introduce a novel approach for optimizing LLM training by adjusting learning rates across weights of different components in Transformer models. Traditional method…
A Survey on Hypothesis Generation for Scientific Discovery in the Era of Large Language Models
Atilla Kaan Alkan, Shashwat Sourav, Maja Jablonska +14
Hypothesis generation is a fundamental step in scientific discovery, yet it is increasingly challenged by information overload and disciplinary fragmentation. Recent advances in La…
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
Jan Ludziejewski, Maciej Pióro, Jakub Krajewski +8
Mixture of Experts (MoE) architectures have significantly increased computational efficiency in both research and real-world applications of large-scale machine learning models. Ho…
Efficient Single-Image Depth Estimation on Mobile Devices, Mobile AI & AIM 2022 Challenge: Report
Andrey Ignatov, Grigory Malivenko, Radu Timofte +36
Various depth estimation models are now widely used on many mobile and IoT devices for image segmentation, bokeh effect rendering, object tracking and many other mobile tasks. Thus…