85 citations · 206 across the 15 of their papers we have counts for
15 papers
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8
Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that…
Learning State-Tracking from Code Using Linear RNNs
Julien Siems, Riccardo Grazzi, Korbinian Pöppel +3
Over the last years, state-tracking tasks, particularly permutation composition, have become a testbed to understand the limits of sequence models architectures like Transformers a…
TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting
Vladyslav Moroshan, Julien Siems, Arber Zela +2
Foundation models for zero-shot time series forecasting face challenges in efficient long-horizon prediction and reproducibility, with existing synthetic-only approaches underperfo…
DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
Julien Siems, Timur Carstensen, Arber Zela +3
Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling, offering efficient training and linear-time inference…
Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
Riccardo Grazzi, Julien Siems, Arber Zela +3
Linear Recurrent Neural Networks (LRNNs) such as Mamba, RWKV, GLA, mLSTM, and DeltaNet have emerged as efficient alternatives to Transformers for long sequences. However, both Tran…
Mamba4Cast: Efficient Zero-Shot Time Series Forecasting with State Space Models
Sathya Kamesh Bhethanabhotla, Omar Swelam, Julien Siems +2
This paper introduces Mamba4Cast, a zero-shot foundation model for time series forecasting. Based on the Mamba architecture and inspired by Prior-data Fitted Networks (PFNs), Mamba…
GAMformer: Bridging Tabular Foundation Models and Interpretable Machine Learning
Andreas Mueller, Julien Siems, Harsha Nori +4
While interpretability is crucial for machine learning applications in safety-critical domains and for regulatory compliance, existing tabular foundation models like TabPFN lack tr…
Is Mamba Capable of In-Context Learning?
Riccardo Grazzi, Julien Siems, Simon Schrodi +2
State of the art foundation models such as GPT-4 perform surprisingly well at in-context learning (ICL), a variant of meta-learning concerning the learned ability to solve tasks du…
Curve Your Enthusiasm: Concurvity Regularization in Differentiable Generalized Additive Models
Julien Siems, Konstantin Ditschuneit, Winfried Ripken +4
Generalized Additive Models (GAMs) have recently experienced a resurgence in popularity due to their interpretability, which arises from expressing the target value as a sum of non…
Interpretable Reinforcement Learning via Neural Additive Models for Inventory Management
Julien Siems, Maximilian Schambach, Sebastian Schulze +1
The COVID-19 pandemic has highlighted the importance of supply chains and the role of digital management to react to dynamic changes in the environment. In this work, we focus on d…
Bayesian Optimization-based Combinatorial Assignment
Jakob Weissteiner, Jakob Heiss, Julien Siems +1
We study the combinatorial assignment domain, which includes combinatorial auctions and course allocation. The main challenge in this domain is that the bundle space grows exponent…
Monotone-Value Neural Networks: Exploiting Preference Monotonicity in Combinatorial Assignment
Jakob Weissteiner, Jakob Heiss, Julien Siems +1
Many important resource allocation problems involve the combinatorial assignment of items, e.g., auctions or course allocation. Because the bundle space grows exponentially in the…
Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks
Arber Zela, Julien Siems, Lucas Zimmer +3
The most significant barrier to the advancement of Neural Architecture Search (NAS) is its demand for large computational resources, which hinders scientifically sound empirical ev…
NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
Arber Zela, Julien Siems, Frank Hutter
One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice. Nevertheless, there is still a lack of understanding…
Faster Training of Mask R-CNN by Focusing on Instance Boundaries
Roland S. Zimmermann, Julien N. Siems
We present an auxiliary task to Mask R-CNN, an instance segmentation network, which leads to faster training of the mask head. Our addition to Mask R-CNN is a new prediction head,…