13 papers
Fast LapSum: Exact Differentiable Top-k at Million Scale
Łukasz Struski, Joanna Wojciechowicz, Jakub Antczak +3
The top- operation is a fundamental building block of modern sparse computation, enabling token routing, expert activation, memory selection, and attention pruning. Yet standard…
LaPrune: Controllable Differentiable Sparsity at Million Scale
Jakub Antczak, Joanna Wojciechowicz, Łukasz Struski +1
Top- selection determines which components of a sparse model remain active. Hard selection blocks gradients, while continuous relaxations often couple mask hardness to the selec…
Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers
Kamil KsiÄ Å¼ek, Piotr SuszyÅski, MichaÅ Jan WÅodarczyk +2
Vision foundation models, such as DINOv2, learn highly expressive representations but rely on massive, opaque architectures that demand substantial computational power and memory.…
What Pixels Are Enough? SEAMS: Sufficiency Saliency via MSE-Preservation Soft-Masks
Magdalena TrÄdowicz, Åukasz Struski, Arkadiusz Lewicki +4
Saliency maps are most useful when they identify the image regions that are sufficient to preserve a model's behaviour. We introduce SEAMS, a sufficiency-based saliency method that…
SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
MikoÅaj Zasada, Åukasz Struski, Jacek Tabor +1
Sparse Mixture-of-Experts (MoE) architectures enable scaling LLM parameters under a fixed inference budget by activating only a small subset of experts via top- routing. While t…
LAPLEX: The FFT of Learnable Laplace Kernels
Åukasz Struski, Hanna Blazhko, Piotr Kubaty +1
Fast linear algebra in deep learning usually comes with a choice: fixed geometry and exact computation, as in the Fourier transform, or adaptive geometry paid for by dense paramete…