5 citations · 9 across the 14 of their papers we have counts for
9 papers · 1 filter
Universal Properties of Activation Sparsity in Modern Large Language Models
Filip Szatkowski, Patryk Będkowski, Alessio Devoto +5
Activation sparsity is an intriguing property of deep neural networks that has been extensively studied in ReLU-based models, due to its advantages for efficiency, robustness, and…
Rethinking Calibration for Early-Exit Neural Networks
Piotr Kubaty, Filip Szatkowski, Grzegorz Choczyński +2
Early-exit neural networks (EENNs) accelerate inference by allowing intermediate classifiers to stop computation once predictions are confident enough. Most methods rely on confide…
Efficient LLM Moderation with Multi-Layer Latent Prototypes
Maciej Chrabąszcz, Filip Szatkowski, Bartosz Wójcik +3
Although modern LLMs are aligned with human values during post-training, robust moderation remains essential to prevent harmful outputs at deployment time. Existing approaches suff…
Exploring the Stability Gap in Continual Learning: The Role of the Classification Head
Wojciech Łapacz, Daniel Marczak, Filip Szatkowski +1
Continual learning (CL) has emerged as a critical area in machine learning, enabling neural networks to learn from evolving data distributions while mitigating catastrophic forgett…
Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initialization
Aleksandra Irena Nowak, Łukasz Gniecki, Filip Szatkowski +1
Static sparse training aims to train sparse models from scratch, achieving remarkable results in recent years. A key design choice is given by the sparse initialization, which dete…
Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
Filip Szatkowski, Yaoyue Zheng, Fei Yang +3
Continual learning is crucial for applying machine learning in challenging, dynamic, and often resource-constrained environments. However, catastrophic forgetting - overwriting pre…
Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion
Filip Szatkowski, Bartosz Wójcik, Mikołaj Piórczyński +1
Transformer models can face practical limitations due to their high computational requirements. At the same time, such models exhibit significant activation sparsity, which can be…
Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual Learning
Filip Szatkowski, Mateusz Pyla, Marcin Przewięźlikowski +3
In this work, we investigate exemplar-free class incremental learning (CIL) with knowledge distillation (KD) as a regularization strategy, aiming to prevent forgetting. KD-based me…
Hypernetworks build Implicit Neural Representations of Sounds
Filip Szatkowski, Karol J. Piczak, Przemysław Spurek +2
Implicit Neural Representations (INRs) are nowadays used to represent multimedia signals across various real-life applications, including image super-resolution, image compression,…