83 citations · 159 across the 10 of their papers we have counts for
11 papers
Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models
Difan Deng, Andreas Bentzen Winje, Lukas Fehring +1
The quadratic computational complexity of softmax transformers has become a bottleneck in long-context scenarios. In contrast, linear attention model families provide a promising d…
carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks
Carolin Benjamins, Helena Graf, Sarah Segel +14
Hyperparameter Optimization (HPO) is crucial to develop well-performing machine learning models. In order to ease prototyping and benchmarking of HPO methods, we propose carps, a b…
Neural Attention Search
Difan Deng, Marius Lindauer
We present Neural Attention Search (NAtS), a framework that automatically evaluates the importance of each token within a sequence and determines if the corresponding token can be…
Optimizing Time Series Forecasting Architectures: A Hierarchical Neural Architecture Search Approach
Difan Deng, Marius Lindauer
The rapid development of time series forecasting research has brought many deep learning-based modules in this field. However, despite the increasing amount of new forecasting arch…
AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks
Alexander Tornede, Difan Deng, Theresa Eimer +8
The fields of both Natural Language Processing (NLP) and Automated Machine Learning (AutoML) have achieved remarkable results over the past years. In NLP, especially Large Language…
Efficient Automated Deep Learning for Time Series Forecasting
Difan Deng, Florian Karl, Frank Hutter +2
Recent years have witnessed tremendously improved efficiency of Automated Machine Learning (AutoML), especially Automated Deep Learning (AutoDL) systems, but recent work focuses on…
Searching in the Forest for Local Bayesian Optimization
Difan Deng, Marius Lindauer
Because of its sample efficiency, Bayesian optimization (BO) has become a popular approach dealing with expensive black-box optimization problems, such as hyperparameter optimizati…
SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
Marius Lindauer, Katharina Eggensperger, Matthias Feurer +6
Algorithm parameters, in particular hyperparameters of machine learning algorithms, can substantially impact their performance. To support users in determining well-performing hype…
Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges
Bernd Bischl, Martin Binder, Michel Lang +9
Most machine learning algorithms are configured by one or several hyperparameters that must be carefully chosen and often considerably impact performance. To avoid a time consuming…
Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization
Julia Guerrero-Viu, Sven Hauns, Sergio Izquierdo +7
Neural architecture search (NAS) and hyperparameter optimization (HPO) make deep learning accessible to non-experts by automatically finding the architecture of the deep neural net…
Squirrel: A Switching Hyperparameter Optimizer
Noor Awad, Gresa Shala, Difan Deng +9
In this short note, we describe our submission to the NeurIPS 2020 BBO challenge. Motivated by the fact that different optimizers work well on different problems, our approach swit…