9 papers
Effective Distillation to Hybrid xLSTM Architectures
Lukas Hauzenberger, Niklas Schmidinger, Thomas Schmied +7
There have been numerous attempts to distill quadratic attention-based large language models (LLMs) into sub-quadratic linearized architectures. However, despite extensive research…
Towards a Neural Debugger for Python
Maximilian Beck, Jonas Gehring, Jannik Kossen +1
Training large language models (LLMs) on Python execution traces grounds them in code execution and enables the line-by-line execution prediction of whole Python programs, effectiv…
xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
Maximilian Beck, Kajetan Schweighofer, Sebastian Böck +2
Scaling laws play a central role in the success of Large Language Models (LLMs), enabling the prediction of model performance relative to compute budgets prior to training. While T…
Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels
Maximilian Beck, Korbinian Pöppel, Phillip Lippe +1
Linear RNNs with gating recently demonstrated competitive performance compared to Transformers in language modeling. Although their linear compute scaling in sequence length offers…
A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks
Thomas Schmied, Thomas Adler, Vihang Patil +6
In recent years, there has been a trend in the field of Reinforcement Learning (RL) towards large action models trained offline on large-scale datasets via sequence modeling. Exist…
xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference
Maximilian Beck, Korbinian Pöppel, Phillip Lippe +5
Recent breakthroughs in solving reasoning, math and coding problems with Large Language Models (LLMs) have been enabled by investing substantial computation budgets at inference ti…