4 papers · 1 filter
KVpop -- Key-Value Cache Compression with Predictive Online Pruning
Lukas Hauzenberger, Niklas Schmidinger, Anamaria-Roberta Hartl +5
Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on…
Effective Distillation to Hybrid xLSTM Architectures
Lukas Hauzenberger, Niklas Schmidinger, Thomas Schmied +7
There have been numerous attempts to distill quadratic attention-based large language models (LLMs) into sub-quadratic linearized architectures. However, despite extensive research…
xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
Maximilian Beck, Kajetan Schweighofer, Sebastian Böck +2
Scaling laws play a central role in the success of Large Language Models (LLMs), enabling the prediction of model performance relative to compute budgets prior to training. While T…
TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning
Andreas Auer, Patrick Podest, Daniel Klotz +3
In-context learning, the ability of large language models to perform tasks using only examples provided in the prompt, has recently been adapted for time series forecasting. This p…