1 citations · 1 across the 5 of their papers we have counts for
5 papers
KVpop -- Key-Value Cache Compression with Predictive Online Pruning
Lukas Hauzenberger, Niklas Schmidinger, Anamaria-Roberta Hartl +5
Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on…
On Subquadratic Architectures: From Applications to Principles
Anamaria-Roberta Hartl, Levente Zólyomi, David Stap +6
Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost. Subquadratic architectures offer a scalable alternative. Howeve…
Effective Distillation to Hybrid xLSTM Architectures
Lukas Hauzenberger, Niklas Schmidinger, Thomas Schmied +7
There have been numerous attempts to distill quadratic attention-based large language models (LLMs) into sub-quadratic linearized architectures. However, despite extensive research…
Parameter Efficient Fine-tuning via Explained Variance Adaptation
Fabian Paischer, Lukas Hauzenberger, Thomas Schmied +3
Foundation models (FMs) are pre-trained on large-scale datasets and then fine-tuned for a specific downstream task. The most common fine-tuning method is to update pretrained weigh…
Modular and On-demand Bias Mitigation with Attribute-Removal Subnetworks
Lukas Hauzenberger, Shahed Masoudian, Deepak Kumar +2
Societal biases are reflected in large pre-trained language models and their fine-tuned versions on downstream tasks. Common in-processing bias mitigation approaches, such as adver…