4 papers
Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models
Difan Deng, Andreas Bentzen Winje, Lukas Fehring +1
The quadratic computational complexity of softmax transformers has become a bottleneck in long-context scenarios. In contrast, linear attention model families provide a promising d…
Neural Attention Search
Difan Deng, Marius Lindauer
We present Neural Attention Search (NAtS), a framework that automatically evaluates the importance of each token within a sequence and determines if the corresponding token can be…
Optimizing Time Series Forecasting Architectures: A Hierarchical Neural Architecture Search Approach
Difan Deng, Marius Lindauer
The rapid development of time series forecasting research has brought many deep learning-based modules in this field. However, despite the increasing amount of new forecasting arch…
carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks
Carolin Benjamins, Helena Graf, Sarah Segel +14
Hyperparameter Optimization (HPO) is crucial to develop well-performing machine learning models. In order to ease prototyping and benchmarking of HPO methods, we propose carps, a b…