6 papers · 1 filter
LoopICL: Looping a single transformer block to solve tabular tasks
Amir Rezaei Balef, Katharina Eggensperger
Tabular foundation models using in-context learning have recently surpassed gradient-boosted trees on predictive tabular tasks. However, recent mechanistic insights suggest that pa…
Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models
Amir Rezaei Balef, Mykhailo Koshil, Katharina Eggensperger
Transformer-based tabular foundation models (TFMs) dominate small to medium tabular predictive benchmark tasks, yet their inference mechanisms remain largely unexplored. We present…
Towards Understanding Layer Contributions in Tabular In-Context Learning Models
Amir Rezaei Balef, Mykhailo Koshil, Katharina Eggensperger
Despite the architectural similarities between tabular in-context learning (ICL) models and large language models (LLMs), little is known about how individual layers contribute to…
TabPFN-Wide: Continued Pre-Training for Extreme Feature Counts
Christopher Kolberg, Jules Kreuer, Jonas Huurdeman +3
Revealing novel insights from the relationship between molecular measurements and pathology remains a very impactful application of machine learning in biomedicine. Data in this do…
In-Context Decision Making for Optimizing Complex AutoML Pipelines
Amir Rezaei Balef, Katharina Eggensperger
Combined Algorithm Selection and Hyperparameter Optimization (CASH) has been fundamental to traditional AutoML systems. However, with the advancements of pre-trained models, modern…
Put CASH on Bandits: A Max K-Armed Problem for Automated Machine Learning
Amir Rezaei Balef, Claire Vernade, Katharina Eggensperger
The Combined Algorithm Selection and Hyperparameter optimization (CASH) is a challenging resource allocation problem in the field of AutoML. We propose MaxUCB, a max k-armed bandit…