4.5k citations · 8.5k across the 90 of their papers we have counts for
74 papers · 1 filter
ParetoTransport: Generative Optimization by Mass Transport Toward The Pareto Front
Stephanie Holly, Sepp Hochreiter, Werner Zellinger
Offline multi-objective optimization requires not only moving the objective vectors of candidate designs toward the Pareto front, but also distributing them effectively along it. G…
KVpop -- Key-Value Cache Compression with Predictive Online Pruning
Lukas Hauzenberger, Niklas Schmidinger, Anamaria-Roberta Hartl +5
Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on…
TiRex-2: Generalizing TiRex to Multivariate Data and Streaming
Patrick Podest, Marco Pichler, Elias Bürger +7
We introduce TiRex-2, a recurrent xLSTM-based time series foundation model that generalizes the univariate TiRex to multivariate forecasting with both past and future covariates. R…
On Subquadratic Architectures: From Applications to Principles
Anamaria-Roberta Hartl, Levente Zólyomi, David Stap +6
Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost. Subquadratic architectures offer a scalable alternative. Howeve…
RREDCoT: Segment-Level Reward Redistribution for Reasoning Models
Mykyta Ielanskyi, Kajetan Schweighofer, Lukas Aichberger +1
Recent advancements in reasoning language models have been driven by Reinforcement Learning (RL) fine-tuning. Most often, these rely on the Group Relative Policy Optimization (GRPO…
Effective Distillation to Hybrid xLSTM Architectures
Lukas Hauzenberger, Niklas Schmidinger, Thomas Schmied +7
There have been numerous attempts to distill quadratic attention-based large language models (LLMs) into sub-quadratic linearized architectures. However, despite extensive research…
Adaptive Retrieval helps Reasoning in LLMs -- but mostly if it's not used
Srijan Shakya, Anamaria-Roberta Hartl, Sepp Hochreiter +1
Large Language Models (LLMs) often falter in complex reasoning tasks due to their static, parametric knowledge, leading to hallucinations and poor performance in specialized domain…
AP-OOD: Attention Pooling for Out-of-Distribution Detection
Claus Hofmann, Christian Huber, Bernhard Lehner +3
Out-of-distribution (OOD) detection, which maps high-dimensional data into a scalar OOD score, is critical for the reliable deployment of machine learning models. A key challenge i…
The Offline-Frontier Shift: Diagnosing Distributional Limits in Generative Multi-Objective Optimization
Stephanie Holly, Alexandru-Ciprian Zăvoianu, Siegfried Silber +2
Offline multi-objective optimization (MOO) aims to recover Pareto-optimal designs given a finite, static dataset. Recent generative approaches, including diffusion models, show str…
Pre-trained Forecasting Models: Strong Zero-Shot Feature Extractors for Time Series Classification
Andreas Auer, Daniel Klotz, Sebastinan Böck +1
Recent research on time series foundation models has primarily focused on forecasting, leaving it unclear how generalizable their learned representations are. In this study, we exa…
Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation
Mykyta Ielanskyi, Kajetan Schweighofer, Lukas Aichberger +1
Hallucinations are a common issue that undermine the reliability of large language models (LLMs). Recent studies have identified a specific subset of hallucinations, known as confa…
xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
Maximilian Beck, Kajetan Schweighofer, Sebastian Böck +2
Scaling laws play a central role in the success of Large Language Models (LLMs), enabling the prediction of model performance relative to compute budgets prior to training. While T…
pLSTM: parallelizable Linear Source Transition Mark networks
Korbinian Pöppel, Richard Freinschlag, Thomas Schmied +2
Modern recurrent architectures, such as xLSTM and Mamba, have recently challenged the Transformer in language modeling. However, their structure constrains their applicability to s…
Rethinking Losses for Diffusion Bridge Samplers
Sebastian Sanokowski, Lukas Gruber, Christoph Bartmann +2
Diffusion bridges are a promising class of deep-learning methods for sampling from unnormalized distributions. Recent works show that the Log Variance (LV) loss consistently outper…
TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning
Andreas Auer, Patrick Podest, Daniel Klotz +3
In-context learning, the ability of large language models to perform tasks using only examples provided in the prompt, has recently been adapted for time series forecasting. This p…
xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference
Maximilian Beck, Korbinian Pöppel, Phillip Lippe +5
Recent breakthroughs in solving reasoning, math and coding problems with Large Language Models (LLMs) have been enabled by investing substantial computation budgets at inference ti…
Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels
Maximilian Beck, Korbinian Pöppel, Phillip Lippe +1
Linear RNNs with gating recently demonstrated competitive performance compared to Transformers in language modeling. Although their linear compute scaling in sequence length offers…
Scalable Discrete Diffusion Samplers: Combinatorial Optimization and Statistical Physics
Sebastian Sanokowski, Wilhelm Berghammer, Martin Ennemoser +3
Learning to sample from complex unnormalized distributions over discrete domains emerged as a promising research direction with applications in statistical physics, variational inf…
Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measure
Lukas Aichberger, Kajetan Schweighofer, Sepp Hochreiter
Large Language Models (LLMs) are increasingly employed in real-world applications, driving the need to evaluate the trustworthiness of their generated text. To this end, reliable u…
FlashRNN: I/O-Aware Optimization of Traditional RNNs on modern hardware
Korbinian Pöppel, Maximilian Beck, Sepp Hochreiter
While Transformers and other sequence-parallelizable neural network architectures seem like the current state of the art in sequence modeling, they specifically lack state-tracking…
A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks
Thomas Schmied, Thomas Adler, Vihang Patil +6
In recent years, there has been a trend in the field of Reinforcement Learning (RL) towards large action models trained offline on large-scale datasets via sequence modeling. Exist…
The Disparate Benefits of Deep Ensembles
Kajetan Schweighofer, Adrian Arnaiz-Rodriguez, Sepp Hochreiter +1
Ensembles of Deep Neural Networks, Deep Ensembles, are widely used as a simple way to boost predictive performance. However, their impact on algorithmic fairness is not well unders…
On Information-Theoretic Measures of Predictive Uncertainty
Kajetan Schweighofer, Lukas Aichberger, Mykyta Ielanskyi +1
Reliable estimation of predictive uncertainty is crucial for machine learning applications, particularly in high-stakes scenarios where hedging against risks is essential. Despite…
Parameter Efficient Fine-tuning via Explained Variance Adaptation
Fabian Paischer, Lukas Hauzenberger, Thomas Schmied +3
Foundation models (FMs) are pre-trained on large-scale datasets and then fine-tuned for a specific downstream task. The most common fine-tuning method is to update pretrained weigh…