23 citations · 33 across the 8 of their papers we have counts for
10 papers
Beyond Local Linearity: Scale-Resolved Geometry of Learned Image Encoders
Jakub Szymkowiak, Wojtek Pałubicki, Kamil Adamczewski
Understanding how learned representations respond to finite input changes is important for characterizing their sensitivity, invariances, and robustness. Yet existing geometric ana…
-Parametrization for Mixture of Experts
Jan Małaśnicki, Kamil Ciebiera, Mateusz Boruń +8
Recent years have seen a growing interest and adoption of LLMs, with Mixture-of-Experts (MoE) emerging as a leading architecture in extremely large models. Currently, the largest o…
Decoupled Relative Learning Rate Schedules
Jan Ludziejewski, Jan Małaśnicki, Maciej Pióro +8
In this work, we introduce a novel approach for optimizing LLM training by adjusting learning rates across weights of different components in Transformer models. Traditional method…
Projected Compression: Trainable Projection for Efficient Transformer Compression
Maciej Stefaniak, Michał Krutul, Jan Małaśnicki +6
Large language models have steadily increased in size to achieve improved performance; however, this growth has also led to greater inference time and computational demands. Conseq…
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
Jan Ludziejewski, Maciej Pióro, Jakub Krajewski +8
Mixture of Experts (MoE) architectures have significantly increased computational efficiency in both research and real-world applications of large-scale machine learning models. Ho…
How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
Piotr Kubaty, Bartosz Wójcik, Bartłomiej Krzepkowski +4
Early exits enable the network's forward pass to terminate early by attaching trainable internal classifiers to the backbone network. Existing early-exit methods typically adopt ei…
AdaGlimpse: Active Visual Exploration with Arbitrary Glimpse Position and Scale
Adam Pardyl, Michał Wronka, Maciej Wołczyk +3
Active Visual Exploration (AVE) is a task that involves dynamically selecting observations (glimpses), which is critical to facilitate comprehension and navigation within an enviro…
Scaling Laws for Fine-Grained Mixture of Experts
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski +9
Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties,…
Revisiting Random Channel Pruning for Neural Network Compression
Yawei Li, Kamil Adamczewski, Wen Li +3
Channel (or 3D filter) pruning serves as an effective way to accelerate the inference of neural networks. There has been a flurry of algorithms that try to solve this practical pro…
DP-MERF: Differentially Private Mean Embeddings with Random Features for Practical Privacy-Preserving Data Generation
Frederik Harder, Kamil Adamczewski, Mijung Park
We propose a differentially private data generation paradigm using random feature representations of kernel mean embeddings when comparing the distribution of true data with that o…