3 citations · 5 across the 6 of their papers we have counts for
8 papers
-Parametrization for Mixture of Experts
Jan Małaśnicki, Kamil Ciebiera, Mateusz Boruń +8
Recent years have seen a growing interest and adoption of LLMs, with Mixture-of-Experts (MoE) emerging as a leading architecture in extremely large models. Currently, the largest o…
Decoupled Relative Learning Rate Schedules
Jan Ludziejewski, Jan Małaśnicki, Maciej Pióro +8
In this work, we introduce a novel approach for optimizing LLM training by adjusting learning rates across weights of different components in Transformer models. Traditional method…
Projected Compression: Trainable Projection for Efficient Transformer Compression
Maciej Stefaniak, Michał Krutul, Jan Małaśnicki +6
Large language models have steadily increased in size to achieve improved performance; however, this growth has also led to greater inference time and computational demands. Conseq…
FlySearch: Exploring how vision-language models explore
Adam Pardyl, Dominik Matuszek, Mateusz Przebieracz +3
The real world is messy and unstructured. Uncovering critical information often requires active, goal-driven exploration. It remains to be seen whether Vision-Language Models (VLMs…
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
Jan Ludziejewski, Maciej Pióro, Jakub Krajewski +8
Mixture of Experts (MoE) architectures have significantly increased computational efficiency in both research and real-world applications of large-scale machine learning models. Ho…
RoboMorph: Evolving Robot Morphology using Large Language Models
Kevin Qiu, Władysław Pałucki, Krzysztof Ciebiera +3
We introduce RoboMorph, an automated approach for generating and optimizing modular robot designs using large language models (LLMs) and evolutionary algorithms. Each robot design…
Scaling Laws for Fine-Grained Mixture of Experts
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski +9
Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties,…
Mixture of Tokens: Continuous MoE through Cross-Example Aggregation
Szymon Antoniak, Michał Krutul, Maciej Pióro +7
Mixture of Experts (MoE) models based on Transformer architecture are pushing the boundaries of language and vision tasks. The allure of these models lies in their ability to subst…