24 citations · 30 across the 9 of their papers we have counts for
10 papers
KaVa: Latent Reasoning via Compressed KV-Cache Distillation
Anna Kuzina, Maciej Pioro, Paul N. Whatmough +1
Large Language Models (LLMs) excel at multi-step reasoning problems with explicit chain-of-thought (CoT), but verbose traces incur significant computational costs and memory overhe…
-Parametrization for Mixture of Experts
Jan Małaśnicki, Kamil Ciebiera, Mateusz Boruń +8
Recent years have seen a growing interest and adoption of LLMs, with Mixture-of-Experts (MoE) emerging as a leading architecture in extremely large models. Currently, the largest o…
Decoupled Relative Learning Rate Schedules
Jan Ludziejewski, Jan Małaśnicki, Maciej Pióro +8
In this work, we introduce a novel approach for optimizing LLM training by adjusting learning rates across weights of different components in Transformer models. Traditional method…
A Survey on Hypothesis Generation for Scientific Discovery in the Era of Large Language Models
Atilla Kaan Alkan, Shashwat Sourav, Maja Jablonska +14
Hypothesis generation is a fundamental step in scientific discovery, yet it is increasingly challenged by information overload and disciplinary fragmentation. Recent advances in La…
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
Jan Ludziejewski, Maciej Pióro, Jakub Krajewski +8
Mixture of Experts (MoE) architectures have significantly increased computational efficiency in both research and real-world applications of large-scale machine learning models. Ho…
State Soup: In-Context Skill Learning, Retrieval and Mixing
Maciej Pióro, Maciej Wołczyk, Razvan Pascanu +2
A new breed of gated-linear recurrent neural networks has reached state-of-the-art performance on a range of sequence modeling problems. Such models naturally handle long sequences…
MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
Maciej Pióro, Kamil Ciebiera, Krystian Król +7
State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers. At the same time, Mixture of Experts (MoE)…
Scaling Laws for Fine-Grained Mixture of Experts
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski +9
Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties,…
Mixture of Tokens: Continuous MoE through Cross-Example Aggregation
Szymon Antoniak, Michał Krutul, Maciej Pióro +7
Mixture of Experts (MoE) models based on Transformer architecture are pushing the boundaries of language and vision tasks. The allure of these models lies in their ability to subst…
Efficient Single-Image Depth Estimation on Mobile Devices, Mobile AI & AIM 2022 Challenge: Report
Andrey Ignatov, Grigory Malivenko, Radu Timofte +36
Various depth estimation models are now widely used on many mobile and IoT devices for image segmentation, bokeh effect rendering, object tracking and many other mobile tasks. Thus…