4.5k citations · 8.5k across the 90 of their papers we have counts for
97 papers
ParetoTransport: Generative Optimization by Mass Transport Toward The Pareto Front
Stephanie Holly, Sepp Hochreiter, Werner Zellinger
Offline multi-objective optimization requires not only moving the objective vectors of candidate designs toward the Pareto front, but also distributing them effectively along it. G…
KVpop -- Key-Value Cache Compression with Predictive Online Pruning
Lukas Hauzenberger, Niklas Schmidinger, Anamaria-Roberta Hartl +5
Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on…
Accelerating LLM Inference via Vector Index Based Output Embeddings
Martin Loretz, Sepp Hochreiter
Large output embedding matrices create a significant memory bandwidth bottleneck during autoregressive decoding, especially for compact LLMs with large multilingual vocabularies. W…
TiRex-2: Generalizing TiRex to Multivariate Data and Streaming
Patrick Podest, Marco Pichler, Elias Bürger +7
We introduce TiRex-2, a recurrent xLSTM-based time series foundation model that generalizes the univariate TiRex to multivariate forecasting with both past and future covariates. R…
On Subquadratic Architectures: From Applications to Principles
Anamaria-Roberta Hartl, Levente Zólyomi, David Stap +6
Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost. Subquadratic architectures offer a scalable alternative. Howeve…
RREDCoT: Segment-Level Reward Redistribution for Reasoning Models
Mykyta Ielanskyi, Kajetan Schweighofer, Lukas Aichberger +1
Recent advancements in reasoning language models have been driven by Reinforcement Learning (RL) fine-tuning. Most often, these rely on the Group Relative Policy Optimization (GRPO…
Unlocking the Working Memory of Large Language Models for Latent Reasoning
Lukas Aichberger, Sepp Hochreiter
To improve the reasoning capabilities of large language models, test-time compute is typically scaled by generating intermediate tokens before the final answer. However, this coupl…
Effective Distillation to Hybrid xLSTM Architectures
Lukas Hauzenberger, Niklas Schmidinger, Thomas Schmied +7
There have been numerous attempts to distill quadratic attention-based large language models (LLMs) into sub-quadratic linearized architectures. However, despite extensive research…
Adaptive Retrieval helps Reasoning in LLMs -- but mostly if it's not used
Srijan Shakya, Anamaria-Roberta Hartl, Sepp Hochreiter +1
Large Language Models (LLMs) often falter in complex reasoning tasks due to their static, parametric knowledge, leading to hallucinations and poor performance in specialized domain…
AP-OOD: Attention Pooling for Out-of-Distribution Detection
Claus Hofmann, Christian Huber, Bernhard Lehner +3
Out-of-distribution (OOD) detection, which maps high-dimensional data into a scalar OOD score, is critical for the reliable deployment of machine learning models. A key challenge i…
The Offline-Frontier Shift: Diagnosing Distributional Limits in Generative Multi-Objective Optimization
Stephanie Holly, Alexandru-Ciprian Zăvoianu, Siegfried Silber +2
Offline multi-objective optimization (MOO) aims to recover Pareto-optimal designs given a finite, static dataset. Recent generative approaches, including diffusion models, show str…
Pre-trained Forecasting Models: Strong Zero-Shot Feature Extractors for Time Series Classification
Andreas Auer, Daniel Klotz, Sebastinan Böck +1
Recent research on time series foundation models has primarily focused on forecasting, leaving it unclear how generalizable their learned representations are. In this study, we exa…
Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation
Mykyta Ielanskyi, Kajetan Schweighofer, Lukas Aichberger +1
Hallucinations are a common issue that undermine the reliability of large language models (LLMs). Recent studies have identified a specific subset of hallucinations, known as confa…
xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
Maximilian Beck, Kajetan Schweighofer, Sebastian Böck +2
Scaling laws play a central role in the success of Large Language Models (LLMs), enabling the prediction of model performance relative to compute budgets prior to training. While T…
Safe and Certifiable AI Systems: Concepts, Challenges, and Lessons Learned
Kajetan Schweighofer, Barbara Brune, Lukas Gruber +11
There is an increasing adoption of artificial intelligence in safety-critical applications, yet practical schemes for certifying that AI systems are safe, lawful and socially accep…
pLSTM: parallelizable Linear Source Transition Mark networks
Korbinian Pöppel, Richard Freinschlag, Thomas Schmied +2
Modern recurrent architectures, such as xLSTM and Mamba, have recently challenged the Transformer in language modeling. However, their structure constrains their applicability to s…
Rethinking Losses for Diffusion Bridge Samplers
Sebastian Sanokowski, Lukas Gruber, Christoph Bartmann +2
Diffusion bridges are a promising class of deep-learning methods for sampling from unnormalized distributions. Recent works show that the Log Variance (LV) loss consistently outper…
TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning
Andreas Auer, Patrick Podest, Daniel Klotz +3
In-context learning, the ability of large language models to perform tasks using only examples provided in the prompt, has recently been adapted for time series forecasting. This p…
xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference
Maximilian Beck, Korbinian Pöppel, Phillip Lippe +5
Recent breakthroughs in solving reasoning, math and coding problems with Large Language Models (LLMs) have been enabled by investing substantial computation budgets at inference ti…
Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels
Maximilian Beck, Korbinian Pöppel, Phillip Lippe +1
Linear RNNs with gating recently demonstrated competitive performance compared to Transformers in language modeling. Although their linear compute scaling in sequence length offers…
Scalable Discrete Diffusion Samplers: Combinatorial Optimization and Statistical Physics
Sebastian Sanokowski, Wilhelm Berghammer, Martin Ennemoser +3
Learning to sample from complex unnormalized distributions over discrete domains emerged as a promising research direction with applications in statistical physics, variational inf…
Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measure
Lukas Aichberger, Kajetan Schweighofer, Sepp Hochreiter
Large Language Models (LLMs) are increasingly employed in real-world applications, driving the need to evaluate the trustworthiness of their generated text. To this end, reliable u…
FlashRNN: I/O-Aware Optimization of Traditional RNNs on modern hardware
Korbinian Pöppel, Maximilian Beck, Sepp Hochreiter
While Transformers and other sequence-parallelizable neural network architectures seem like the current state of the art in sequence modeling, they specifically lack state-tracking…
Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences
Niklas Schmidinger, Lisa Schneckenreiter, Philipp Seidl +7
Language models for biological and chemical sequences enable crucial applications such as drug discovery, protein engineering, and precision medicine. Currently, these language mod…