5 citations · 12 across the 21 of their papers we have counts for
22 papers
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8
Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that…
TimEE: End-to-end Time Series Classification via In-Context Learning
Jaris Küken, Shi Bin Hoo, Martin Mráz +2
Time series classification (TSC) is dominated by a two-stage paradigm: train a feature encoder -- either from scratch on the target dataset or via pretraining on large corpora -- a…
Towards Evaluating Data Priors for Tabular Foundation Models
Zeynep Türkmen, Kürşat Kaya, Alexander Pfefferle +1
Data-generating priors are a central component of tabular foundation models because they define the task distribution used during pretraining. However, priors are rarely evaluated…
Towards Pretraining Text Encoders for TabPFN
Mustafa Tajjar, Alexander Pfefferle, Lennart Purucker +1
Tabular foundation models, such as TabPFN, achieve strong performance on tabular datasets with numerical and categorical data, but do not natively handle high-cardinality text feat…
Causal Data Augmentation for Robust Fine-Tuning of Tabular Foundation Models
Magnus Bühler, Lennart Purucker, Frank Hutter
Fine-tuning tabular foundation models (TFMs) under data scarcity is challenging, as early stopping on even scarcer validation data often fails to capture true generalization perfor…
nanoTabPFN: A Lightweight and Educational Reimplementation of TabPFN
Alexander Pfefferle, Johannes Hog, Lennart Purucker +1
Tabular foundation models such as TabPFN have revolutionized predictive machine learning for tabular data. At the same time, the driving factors of this revolution are hard to unde…
Does TabPFN Understand Causal Structures?
Omar Swelam, Lennart Purucker, Jake Robertson +3
Causal discovery is fundamental for multiple scientific domains, yet extracting causal information from real world data remains a significant challenge. Given the recent success on…
Selective Rotary Position Embedding
Sajad Movahedi, Timur Carstensen, Arshia Afzal +3
Position information is essential for language modeling. In softmax transformers, Rotary Position Embeddings (\textit{RoPE}) encode positions through \textit{fixed-angle} rotations…
TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models
Léo Grinsztajn, Klemens Flöge, Oscar Key +23
The first tabular foundation model, TabPFN, and its successor TabPFNv2 have impacted tabular AI substantially, with dozens of methods building on it and hundreds of applications ac…
Agentic NL2SQL to Reduce Computational Costs
Dominik Jehle, Lennart Purucker, Frank Hutter
Translating natural language queries into SQL queries (NL2SQL or Text-to-SQL) has recently been empowered by large language models (LLMs). Using LLMs to perform NL2SQL methods on a…
TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting
Vladyslav Moroshan, Julien Siems, Arber Zela +2
Foundation models for zero-shot time series forecasting face challenges in efficient long-horizon prediction and reproducibility, with existing synthetic-only approaches underperfo…
Quickly Tuning Foundation Models for Image Segmentation
Breenda Das, Lennart Purucker, Timur Carstensen +1
Foundation models like SAM (Segment Anything Model) exhibit strong zero-shot image segmentation performance, but often fall short on domain-specific tasks. Fine-tuning these models…
How Usable is Automated Feature Engineering for Tabular Data?
Bastian Schäfer, Lennart Purucker, Maciej Janowski +1
Tabular data, consisting of rows and columns, is omnipresent across various machine learning applications. Each column represents a feature, and features can be combined or transfo…
Early Stopping Tabular In-Context Learning
Jaris Küken, Lennart Purucker, Frank Hutter
Tabular foundation models have shown strong performance across various tabular learning tasks via in-context learning, offering robust generalization without any downstream finetun…
Unreflected Use of Tabular Data Repositories Can Undermine Research Quality
Andrej Tschalzev, Lennart Purucker, Stefan Lüdtke +3
Data repositories have accumulated a large number of tabular datasets from various domains. Machine Learning researchers are actively using these datasets to evaluate novel approac…
DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
Julien Siems, Timur Carstensen, Arber Zela +3
Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling, offering efficient training and linear-time inference…
Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
Riccardo Grazzi, Julien Siems, Arber Zela +3
Linear Recurrent Neural Networks (LRNNs) such as Mamba, RWKV, GLA, mLSTM, and DeltaNet have emerged as efficient alternatives to Transformers for long sequences. However, both Tran…
Ensembling Finetuned Language Models for Text Classification
Sebastian Pineda Arango, Maciej Janowski, Lennart Purucker +3
Finetuning is a common practice widespread across different communities to adapt pretrained models to particular tasks. Text classification is one of these tasks for which many pre…
Mamba4Cast: Efficient Zero-Shot Time Series Forecasting with State Space Models
Sathya Kamesh Bhethanabhotla, Omar Swelam, Julien Siems +2
This paper introduces Mamba4Cast, a zero-shot foundation model for time series forecasting. Based on the Mamba architecture and inspired by Prior-data Fitted Networks (PFNs), Mamba…
Large Language Models Engineer Too Many Simple Features For Tabular Data
Jaris Küken, Lennart Purucker, Frank Hutter
Tabular machine learning problems often require time-consuming and labor-intensive feature engineering. Recent efforts have focused on using large language models (LLMs) to capital…
Regularized Neural Ensemblers
Sebastian Pineda Arango, Maciej Janowski, Lennart Purucker +3
Ensemble methods are known for enhancing the accuracy and robustness of machine learning models by combining multiple base learners. However, standard approaches like greedy or ran…
Is Mamba Capable of In-Context Learning?
Riccardo Grazzi, Julien Siems, Simon Schrodi +2
State of the art foundation models such as GPT-4 perform surprisingly well at in-context learning (ICL), a variant of meta-learning concerning the learned ability to solve tasks du…