Publications (213)
Convergence Analysis of Homotopy-SGD for non-convex optimization
Matilde Gargiani, Andrea Zanelli, Quoc Tran-Dinh +2
First-order stochastic methods for solving large-scale non-convex optimization problems are widely used in many big-data applications, e.g. training deep neural networks as well as…
Towards White-box Benchmarks for Algorithm Control
André Biedenkapp, H. Furkan Bozkurt, Frank Hutter +1
The performance of many algorithms in the fields of hard combinatorial problem solving, machine learning or AI in general depends on tuned hyperparameter configurations. Automated…
Warmstarting of Model-based Algorithm Configuration
Marius Lindauer, Frank Hutter
The performance of many hard combinatorial problem solvers depends strongly on their parameter settings, and since manual parameter tuning is both tedious and suboptimal the AI com…
Scalable Deep Learning for RNA Secondary Structure Prediction
Jörg K. H. Franke, Frederic Runge, Frank Hutter
The field of RNA secondary structure prediction has made significant progress with the adoption of deep learning techniques. In this work, we present the RNAformer, a lean deep lea…
NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
Arber Zela, Julien Siems, Frank Hutter
One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice. Nevertheless, there is still a lack of understanding…
Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari
Patryk Chrabaszcz, Ilya Loshchilov, Frank Hutter
Evolution Strategies (ES) have recently been demonstrated to be a viable alternative to reinforcement learning (RL) algorithms on a set of challenging deep RL problems, including A…
Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars
Simon Schrodi, Danny Stoll, Binxin Ru +3
The discovery of neural architectures from simple building blocks is a long-standing goal of Neural Architecture Search (NAS). Hierarchical search spaces are a promising step towar…
A case study of algorithm selection for the traveling thief problem
Markus Wagner, Marius Lindauer, Mustafa Misir +2
Many real-world problems are composed of several interacting components. In order to facilitate research on such interactions, the Traveling Thief Problem (TTP) was created in 2013…
Bayesian Optimization in a Billion Dimensions via Random Embeddings
Ziyu Wang, Frank Hutter, Masrour Zoghi +2
Bayesian optimization techniques have been successfully applied to robotics, planning, sensor placement, recommendation, advertising, intelligent user interfaces and automatic algo…
Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces
Kevin Swersky, David Duvenaud, Jasper Snoek +2
In practical Bayesian optimization, we must often search over structures with differing numbers of parameters. For instance, we may wish to search over neural network architectures…
Learning in Compact Spaces with Approximately Normalized Transformer
Jörg K. H. Franke, Urs Spiegelhalter, Marianna Nezhurina +3
The successful training of deep neural networks requires addressing challenges such as overfitting, numerical instabilities leading to divergence, and increasing variance in the re…
The Configurable SAT Solver Challenge (CSSC)
Frank Hutter, Marius Lindauer, Adrian Balint +3
It is well known that different solution strategies work well for different types of instances of hard combinatorial problems. As a consequence, most solvers for the propositional…
Tuning LLM Judge Design Decisions for 1/1000 of the Cost
David Salinas, Omar Swelam, Frank Hutter
Evaluating Large Language Models (LLMs) often requires costly human annotations. To address this, LLM-based judges have been proposed, which compare the outputs of two LLMs enablin…
Towards Benchmarking Foundation Models for Tabular Data With Text
Martin Mráz, Breenda Das, Anshul Gupta +2
Foundation models for tabular data are rapidly evolving, with increasing interest in extending them to support additional modalities such as free-text features. However, existing b…
DEHB: Evolutionary Hyperband for Scalable, Robust and Efficient Hyperparameter Optimization
Noor Awad, Neeratyoy Mallik, Frank Hutter
Modern machine learning algorithms crucially rely on several design decisions to achieve strong performance, making the problem of Hyperparameter Optimization (HPO) more important…
confopt: A Library for Implementation and Evaluation of Gradient-based One-Shot NAS Methods
Abhash Kumar Jha, Shakiba Moradian, Arjun Krishnakumar +2
Gradient-based one-shot neural architecture search (NAS) has significantly reduced the cost of exploring architectural spaces with discrete design choices, such as selecting operat…
Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
Arber Zela, Aaron Klein, Stefan Falkner +1
While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter…
A Kernel for Hierarchical Parameter Spaces
Frank Hutter, Michael A. Osborne
We define a family of kernels for mixed continuous/discrete hierarchical parameter spaces and show that they are positive definite.
Tune My Adam, Please!
Theodoros Athanasiadis, Steven Adriaensen, Samuel Müller +1
The Adam optimizer remains one of the most widely used optimizers in deep learning, and effectively tuning its hyperparameters is key to optimizing performance. However, tuning can…
Towards Automated Design of Riboswitches
Frederic Runge, Jörg K. H. Franke, Frank Hutter
Experimental screening and selection pipelines for the discovery of novel riboswitches are expensive, time-consuming, and inefficient. Using computational methods to reduce the num…
Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
Jack Parker-Holder, Raghu Rajan, Xingyou Song +9
The combination of Reinforcement Learning (RL) with deep learning has led to a series of impressive feats, with many believing (deep) RL provides a path towards generally capable a…
Beyond Random Augmentations: Pretraining with Hard Views
Fabio Ferreira, Ivo Rapant, Jörg K. H. Franke +1
Self-Supervised Learning (SSL) methods typically rely on random image augmentations, or views, to make models invariant to different transformations. We hypothesize that the effica…
Transformers Can Do Bayesian Inference
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango +2
Currently, it is hard to reap the benefits of deep learning for Bayesian methods, which allow the explicit specification of prior knowledge and accurately capture model uncertainty…
Position: A Call to Action for a Human-Centered AutoML Paradigm
Marius Lindauer, Florian Karl, Anne Klier +6
Automated machine learning (AutoML) was formed around the fundamental objectives of automatically and efficiently configuring machine learning (ML) workflows, aiding the research o…
MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image
Alan Arazi, Eilam Shapira, Shoham Grunblat +8
Tabular Foundation Models have recently established the state of the art in supervised tabular learning, by leveraging pretraining to learn generalizable representations of numeric…
A Llama walks into the 'Bar': Efficient Supervised Fine-Tuning for Legal Reasoning in the Multi-state Bar Exam
Rean Fernandes, André Biedenkapp, Frank Hutter +1
Legal reasoning tasks present unique challenges for large language models (LLMs) due to the complexity of domain-specific knowledge and reasoning processes. This paper investigates…
HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models
Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler +4
The increasing size of language models necessitates a thorough analysis across multiple dimensions to assess trade-offs among crucial hardware metrics such as latency, energy consu…
Machine-Learning-Based Diagnostics of EEG Pathology
Lukas Alexander Wilhelm Gemein, Robin Tibor Schirrmeister, Patryk ChrabÄ szcz +5
Machine learning (ML) methods have the potential to automate clinical EEG analysis. They can be categorized into feature-based (with handcrafted features), and end-to-end approache…
Smooth Variational Graph Embeddings for Efficient Neural Architecture Search
Jovita Lukasik, David Friede, Arber Zela +2
Neural architecture search (NAS) has recently been addressed from various directions, including discrete, sampling-based methods and efficient differentiable approaches. While the…
Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data
Kai Helli, David Schnurr, Noah Hollmann +2
While most ML models expect independent and identically distributed data, this assumption is often violated in real-world scenarios due to distribution shifts, resulting in the deg…
TabPFN-3: Technical Report
Léo Grinsztajn, Klemens Flöge, Oscar Key +38
Tabular data underpins most high-value prediction problems in science and industry, and TabPFN has driven the foundation model revolution for this modality. Designed with feedback…
Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization
Michael Volpp, Lukas P. Fröhlich, Kirsten Fischer +4
Transferring knowledge across tasks to improve data-efficiency is one of the open key challenges in the field of global black-box optimization. Readily available algorithms are typ…
Meta-Surrogate Benchmarking for Hyperparameter Optimization
Aaron Klein, Zhenwen Dai, Frank Hutter +2
Despite the recent progress in hyperparameter optimization (HPO), available benchmarks that resemble real-world scenarios consist of a few and very large problem instances that are…
Do-PFN: In-Context Learning for Causal Effect Estimation
Jake Robertson, Arik Reuter, Siyuan Guo +3
Estimation of causal effects is critical to a range of scientific disciplines. Existing methods for this task either require interventional data, knowledge about the ground truth c…
Neural Architecture Search: Insights from 1000 Papers
Colin White, Mahmoud Safari, Rhea Sukthanker +5
In the past decade, advances in deep learning have resulted in breakthroughs in a variety of areas, including computer vision, natural language understanding, speech recognition, a…
OpenML-Python: an extensible Python API for OpenML
Matthias Feurer, Jan N. van Rijn, Arlind Kadra +6
OpenML is an online platform for open science collaboration in machine learning, used to share datasets and results of machine learning experiments. In this paper we introduce Open…
Squirrel: A Switching Hyperparameter Optimizer
Noor Awad, Gresa Shala, Difan Deng +9
In this short note, we describe our submission to the NeurIPS 2020 BBO challenge. Motivated by the fact that different optimizers work well on different problems, our approach swit…
Unreflected Use of Tabular Data Repositories Can Undermine Research Quality
Andrej Tschalzev, Lennart Purucker, Stefan Lüdtke +3
Data repositories have accumulated a large number of tabular datasets from various domains. Machine Learning researchers are actively using these datasets to evaluate novel approac…
carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks
Carolin Benjamins, Helena Graf, Sarah Segel +14
Hyperparameter Optimization (HPO) is crucial to develop well-performing machine learning models. In order to ease prototyping and benchmarking of HPO methods, we propose carps, a b…
Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics
Indrashis Das, Mahmoud Safari, Steven Adriaensen +1
Activation functions are fundamental elements of deep learning architectures as they significantly influence training dynamics. ReLU, while widely used, is prone to the dying neuro…
Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms
Chris Thornton, Frank Hutter, Holger H. Hoos +1
Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall. We co…
Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
Dongwoo Lee, Dong Bok Lee, Steven Adriaensen +5
Scaling has been a major driver of recent advancements in deep learning. Numerous empirical studies have found that scaling laws often follow the power-law and proposed several var…
PFNs4BO: In-Context Learning for Bayesian Optimization
Samuel Müller, Matthias Feurer, Noah Hollmann +1
In this paper, we use Prior-data Fitted Networks (PFNs) as a flexible surrogate for Bayesian Optimization (BO). PFNs are neural processes that are trained to approximate the poster…
HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik +6
To achieve peak predictive performance, hyperparameter optimization (HPO) is a crucial component of machine learning and its applications. Over the last years, the number of effici…
Automated Dynamic Algorithm Configuration
Steven Adriaensen, André Biedenkapp, Gresa Shala +4
The performance of an algorithm often critically depends on its parameter configuration. While a variety of automated algorithm configuration methods have been proposed to relieve…
Learning to Design RNA
Frederic Runge, Danny Stoll, Stefan Falkner +1
Designing RNA molecules has garnered recent interest in medicine, synthetic biology, biotechnology and bioinformatics since many functional RNA molecules were shown to be involved…
Contextualize Me -- The Case for Context in Reinforcement Learning
Carolin Benjamins, Theresa Eimer, Frederik Schubert +6
While Reinforcement Learning ( RL) has made great strides towards solving increasingly complicated problems, many algorithms are still brittle to even slight environmental changes.…
Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
Sheheryar Zaidi, Arber Zela, Thomas Elsken +3
Ensembles of neural networks achieve superior performance compared to stand-alone networks in terms of accuracy, uncertainty calibration and robustness to dataset shift. \emph{Deep…
Increasing LLM Coding Capabilities through Diverse Synthetic Coding Tasks
Amal Abed, Ivan Lukic, Jörg K. H. Franke +1
Large language models (LLMs) have shown impressive promise in code generation, yet their progress remains limited by the shortage of large-scale datasets that are both diverse and…
Bag of Tricks for Neural Architecture Search
Thomas Elsken, Benedikt Staffler, Arber Zela +2
While neural architecture search methods have been successful in previous years and led to new state-of-the-art performance on various problems, they have also been criticized for…
Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters
Marius Lindauer, Matthias Feurer, Katharina Eggensperger +2
Bayesian Optimization (BO) is a common approach for hyperparameter optimization (HPO) in automated machine learning. Although it is well-accepted that HPO is crucial to obtain well…
OpenML Benchmarking Suites
Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer +6
Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks. We advocate the use of curated, comprehensive suites of machine l…
Is Mamba Capable of In-Context Learning?
Riccardo Grazzi, Julien Siems, Simon Schrodi +2
State of the art foundation models such as GPT-4 perform surprisingly well at in-context learning (ICL), a variant of meta-learning concerning the learned ability to solve tasks du…
Beyond IID: How General Are Tabular Foundation Models, Really?
Lennart Purucker, Andrej Tschalzev, Nick Erickson +7
Foundation models for predictive machine learning on tabular data have recently gained significant traction in academia and industry. Research communities across disciplines are in…
STRABLE: Benchmarking Tabular Machine Learning with Strings
Gioia Blayer, Myung Jun Kim, Félix Lefebvre +8
Benchmarking tabular learning has revealed the benefit of dedicated architectures, pushing the state of the art. But real-world tables often contain string entries, beyond numbers,…
nanoTabPFN: A Lightweight and Educational Reimplementation of TabPFN
Alexander Pfefferle, Johannes Hog, Lennart Purucker +1
Tabular foundation models such as TabPFN have revolutionized predictive machine learning for tabular data. At the same time, the driving factors of this revolution are hard to unde…
Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification
Adrian El Baz, Ihsan Ullah, Edesio Alcobaça +17
Although deep neural networks are capable of achieving performance superior to humans on various tasks, they are notorious for requiring large amounts of data and computing resourc…
LMEMs for post-hoc analysis of HPO Benchmarking
Anton Geburek, Neeratyoy Mallik, Danny Stoll +2
The importance of tuning hyperparameters in Machine Learning (ML) and Deep Learning (DL) is established through empirical research and applications, evident from the increase in ne…
Managing extreme AI risks amid rapid progress
Yoshua Bengio, Geoffrey Hinton, Andrew Yao +22
Artificial Intelligence (AI) is progressing rapidly, and companies are shifting their focus to developing generalist AI systems that can autonomously act and pursue goals. Increase…
Well-tuned Simple Nets Excel on Tabular Datasets
Arlind Kadra, Marius Lindauer, Frank Hutter +1
Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent…
ParamILS: An Automatic Algorithm Configuration Framework
Frank Hutter, Thomas Stuetzle, Kevin Leyton-Brown +1
The identification of performance-optimizing parameter settings is an important part of the development and application of algorithms. We describe an automatic framework for this a…
A Human-in-the-Loop Fairness-Aware Model Selection Framework for Complex Fairness Objective Landscapes
Jake Robertson, Thorsten Schmidt, Frank Hutter +1
Fairness-aware Machine Learning (FairML) applications are often characterized by complex social objectives and legal requirements, frequently involving multiple, potentially confli…
Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Data
Anurag Garg, Muhammad Ali, Noah Hollmann +3
Foundation models for tabular data, like TabPFN, achieve strong performance on small datasets when pre-trained solely on synthetic data. We show that this performance can be signif…
Simple And Efficient Architecture Search for Convolutional Neural Networks
Thomas Elsken, Jan-Hendrik Metzen, Frank Hutter
Neural networks have recently had a lot of success for many tasks. However, neural network architectures that perform well are still typically designed manually by experts in a cum…
Position: The Future of Bayesian Prediction Is Prior-Fitted
Samuel Müller, Arik Reuter, Noah Hollmann +2
Training neural networks on randomly generated artificial datasets yields Bayesian models that capture the prior defined by the dataset-generating distribution. Prior-data Fitted N…
Probabilistic Transformer: Modelling Ambiguities and Distributions for RNA Folding and Molecule Design
Jörg K. H. Franke, Frederic Runge, Frank Hutter
Our world is ambiguous and this is reflected in the data we use to train our algorithms. This is particularly true when we try to model natural processes where collected data is af…
ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning
Jannis Becktepe, Julian Dierkes, Carolin Benjamins +7
Hyperparameters are a critical factor in reliably training well-performing reinforcement learning (RL) agents. Unfortunately, developing and evaluating automated approaches for tun…
Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarks
Shuhei Watanabe, Neeratyoy Mallik, Edward Bergman +1
While deep learning has celebrated many successes, its results often hinge on the meticulous selection of hyperparameters (HPs). However, the time-consuming nature of deep learning…
In-Loop Meta-Learning with Gradient-Alignment Reward
Samuel Müller, André Biedenkapp, Frank Hutter
At the heart of the standard deep learning training loop is a greedy gradient step minimizing a given loss. We propose to add a second step to maximize training generalization. To…
Neural Model-based Optimization with Right-Censored Observations
Katharina Eggensperger, Kai Haase, Philipp Müller +2
In many fields of study, we only observe lower bounds on the true response value of some experiments. When fitting a regression model to predict the distribution of the outcomes, w…
Online Batch Selection for Faster Training of Neural Networks
Ilya Loshchilov, Frank Hutter
Deep neural networks are commonly trained using stochastic non-convex optimization procedures, which are driven by gradient information estimated on fractions (batches) of the data…
CMA-ES for Hyperparameter Optimization of Deep Neural Networks
Ilya Loshchilov, Frank Hutter
Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization. As an alternative, we propose to use the Covariance Matrix Adapt…
Multi-headed Neural Ensemble Search
Ashwin Raaghav Narayanan, Arber Zela, Tonmoy Saikia +2
Ensembles of CNN models trained with different seeds (also known as Deep Ensembles) are known to achieve superior performance over a single copy of the CNN. Neural Ensemble Search…
Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How
Sebastian Pineda Arango, Fabio Ferreira, Arlind Kadra +2
With the ever-increasing number of pretrained models, machine learning practitioners are continuously faced with which pretrained model to use, and how to finetune it for a new dat…
A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
Patryk Chrabaszcz, Ilya Loshchilov, Frank Hutter
The original ImageNet dataset is a popular large-scale benchmark for training Deep Neural Networks. Since the cost of performing experiments (e.g, algorithm design, architecture se…
SurvPFN: Towards Foundation Models for Survival Predictions
Samuel Böhm, Lennart Purucker, Frank Hutter +1
Tabular foundation models (TFMs) have made rapid progress in standard classification and regression, but time-to-event survival prediction tasks have remained largely untouched. Un…
Asynchronous Stochastic Gradient MCMC with Elastic Coupling
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner +1
We consider parallel asynchronous Markov Chain Monte Carlo (MCMC) sampling for problems where we can leverage (stochastic) gradients to define continuous dynamics which explore the…
Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
Aaron Klein, Frank Hutter
Due to the high computational demands executing a rigorous comparison between hyperparameter optimization (HPO) methods is often cumbersome. The goal of this paper is to facilitate…
How Usable is Automated Feature Engineering for Tabular Data?
Bastian Schäfer, Lennart Purucker, Maciej Janowski +1
Tabular data, consisting of rows and columns, is omnipresent across various machine learning applications. Each column represents a feature, and features can be combined or transfo…
One-shot World Models Using a Transformer Trained on a Synthetic Prior
Fabio Ferreira, Moreno Schlageter, Raghu Rajan +2
A World Model is a compressed spatial and temporal representation of a real world environment that allows one to train an agent or execute planning methods. However, world models a…
Practical Transfer Learning for Bayesian Optimization
Matthias Feurer, Benjamin Letham, Frank Hutter +1
When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset. We…
On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
Baohe Zhang, Raghu Rajan, Luis Pineda +5
Model-based Reinforcement Learning (MBRL) is a promising framework for learning control in a data-efficient manner. MBRL algorithms can be fairly complex due to the separate dynami…
Self-Paced Context Evaluation for Contextual Reinforcement Learning
Theresa Eimer, André Biedenkapp, Frank Hutter +1
Reinforcement learning (RL) has made a lot of advances for solving a single problem in a given environment; but learning policies that generalize to unseen variations of a problem…
PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning
Neeratyoy Mallik, Edward Bergman, Carl Hvarfner +5
Hyperparameters of Deep Learning (DL) pipelines are crucial for their downstream performance. While a large number of methods for Hyperparameter Optimization (HPO) have been develo…
Hyperparameter Transfer Across Developer Adjustments
Danny Stoll, Jörg K. H. Franke, Diane Wagner +2
After developer adjustments to a machine learning (ML) algorithm, how can the results of an old hyperparameter optimization (HPO) automatically be used to speedup a new HPO? This q…
Bayes' Power for Explaining In-Context Learning Generalizations
Samuel Müller, Noah Hollmann, Frank Hutter
Traditionally, neural network training has been primarily viewed as an approximation of maximum likelihood estimation (MLE). This interpretation originated in a time when training…
Mind the Gap: Measuring Generalization Performance Across Multiple Objectives
Matthias Feurer, Katharina Eggensperger, Edward Bergman +3
Modern machine learning models are often constructed taking into account multiple objectives, e.g., minimizing inference time while also maximizing accuracy. Multi-objective hyperp…
Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition
Samuel Dooley, Rhea Sanjay Sukthanker, John P. Dickerson +3
Face recognition systems are widely deployed in safety-critical applications, including law enforcement, yet they exhibit bias across a range of socio-demographic dimensions, such…
NAS-Bench-101: Towards Reproducible Neural Architecture Search
Chris Ying, Aaron Klein, Esteban Real +3
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to r…
Rethinking Performance Measures of RNA Secondary Structure Problems
Frederic Runge, Jörg K. H. Franke, Daniel Fertmann +1
Accurate RNA secondary structure prediction is vital for understanding cellular regulation and disease mechanisms. Deep learning (DL) methods have surpassed traditional algorithms…
Multi-Objective Hierarchical Optimization with Large Language Models
Andrej Schwanke, Lyubomir Ivanov, David Salinas +2
Despite their widespread adoption in various domains, especially due to their powerful reasoning capabilities, Large Language Models (LLMs) are not the off-the-shelf choice to driv…
Multi-objective Differentiable Neural Architecture Search
Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler +3
Pareto front profiling in multi-objective optimization (MOO), i.e., finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives that req…
Agentic NL2SQL to Reduce Computational Costs
Dominik Jehle, Lennart Purucker, Frank Hutter
Translating natural language queries into SQL queries (NL2SQL or Text-to-SQL) has recently been empowered by large language models (LLMs). Using LLMs to perform NL2SQL methods on a…
Multi-objective Hyperparameter Optimization in the Age of Deep Learning
Soham Basu, Frank Hutter, Danny Stoll
While Deep Learning (DL) experts often have prior knowledge about which hyperparameter settings yield strong performance, only few Hyperparameter Optimization (HPO) algorithms can…
Selective Rotary Position Embedding
Sajad Movahedi, Timur Carstensen, Arshia Afzal +3
Position information is essential for language modeling. In softmax transformers, Rotary Position Embeddings (\textit{RoPE}) encode positions through \textit{fixed-angle} rotations…
FairPFN: Transformers Can do Counterfactual Fairness
Jake Robertson, Noah Hollmann, Noor Awad +1
Machine Learning systems are increasingly prevalent across healthcare, law enforcement, and finance but often operate on historical data, which may carry biases against certain dem…
Quickly Tuning Foundation Models for Image Segmentation
Breenda Das, Lennart Purucker, Timur Carstensen +1
Foundation models like SAM (Segment Anything Model) exhibit strong zero-shot image segmentation performance, but often fall short on domain-specific tasks. Fine-tuning these models…
Best Practices for Scientific Research on Neural Architecture Search
Marius Lindauer, Frank Hutter
Finding a well-performing architecture is often tedious for both DL practitioners and researchers, leading to tremendous interest in the automation of this task by means of neural…
Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization
Timur Carstensen, Neeratyoy Mallik, Frank Hutter +1
As model sizes grow, finding efficient and cost-effective hyperparameter optimization (HPO) methods becomes increasingly crucial for deep learning pipelines. While multi-fidelity H…
Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization
Julia Guerrero-Viu, Sven Hauns, Sergio Izquierdo +7
Neural architecture search (NAS) and hyperparameter optimization (HPO) make deep learning accessible to non-experts by automatically finding the architecture of the deep neural net…