262 citations · 596 across the 23 of their papers we have counts for
24 papers
Evolving Many Worlds: Towards Open-Ended Discovery in Petri Dish NCA via Population-Based Training
Uljad Berdica, Jakob Foerster, Frank Hutter +1
The generation of sustained, open-ended complexity from local interactions remains a fundamental challenge in artificial life. Differentiable multi-agent systems, such as Petri Dis…
Multi-Objective Hierarchical Optimization with Large Language Models
Andrej Schwanke, Lyubomir Ivanov, David Salinas +2
Despite their widespread adoption in various domains, especially due to their powerful reasoning capabilities, Large Language Models (LLMs) are not the off-the-shelf choice to driv…
TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting
Vladyslav Moroshan, Julien Siems, Arber Zela +2
Foundation models for zero-shot time series forecasting face challenges in efficient long-horizon prediction and reproducibility, with existing synthetic-only approaches underperfo…
Improving LLM-based Global Optimization with Search Space Partitioning
Andrej Schwanke, Lyubomir Ivanov, David Salinas +4
Large Language Models (LLMs) have recently emerged as effective surrogate models and candidate generators within global optimization frameworks for expensive blackbox functions. De…
DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
Julien Siems, Timur Carstensen, Arber Zela +3
Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling, offering efficient training and linear-time inference…
Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
Riccardo Grazzi, Julien Siems, Arber Zela +3
Linear Recurrent Neural Networks (LRNNs) such as Mamba, RWKV, GLA, mLSTM, and DeltaNet have emerged as efficient alternatives to Transformers for long sequences. However, both Tran…
Ensembling Finetuned Language Models for Text Classification
Sebastian Pineda Arango, Maciej Janowski, Lennart Purucker +3
Finetuning is a common practice widespread across different communities to adapt pretrained models to particular tasks. Text classification is one of these tasks for which many pre…
Regularized Neural Ensemblers
Sebastian Pineda Arango, Maciej Janowski, Lennart Purucker +3
Ensemble methods are known for enhancing the accuracy and robustness of machine learning models by combining multiple base learners. However, standard approaches like greedy or ran…
HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models
Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler +4
The increasing size of language models necessitates a thorough analysis across multiple dimensions to assess trade-offs among crucial hardware metrics such as latency, energy consu…
Multi-objective Differentiable Neural Architecture Search
Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler +3
Pareto front profiling in multi-objective optimization (MOO), i.e., finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives that req…
Neural Architecture Search: Insights from 1000 Papers
Colin White, Mahmoud Safari, Rhea Sukthanker +5
In the past decade, advances in deep learning have resulted in breakthroughs in a variety of areas, including computer vision, natural language understanding, speech recognition, a…
NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies
Arjun Krishnakumar, Colin White, Arber Zela +3
Zero-cost proxies (ZC proxies) are a recent architecture performance prediction technique aiming to significantly speed up algorithms for neural architecture search (NAS). Recent w…
NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy
Yash Mehta, Colin White, Arber Zela +6
The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-201, has significantly lowered the computational overhead for conducting scientific research in neural archit…
Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019
Zhengying Liu, Adrien Pavao, Zhen Xu +22
This paper reports the results and post-challenge analyses of ChaLearn's AutoDL challenge series, which helped sorting out a profusion of AutoML solutions for Deep Learning (DL) th…
Multi-headed Neural Ensemble Search
Ashwin Raaghav Narayanan, Arber Zela, Tonmoy Saikia +2
Ensembles of CNN models trained with different seeds (also known as Deep Ensembles) are known to achieve superior performance over a single copy of the CNN. Neural Ensemble Search…
Bag of Tricks for Neural Architecture Search
Thomas Elsken, Benedikt Staffler, Arber Zela +2
While neural architecture search methods have been successful in previous years and led to new state-of-the-art performance on various problems, they have also been criticized for…
How Powerful are Performance Predictors in Neural Architecture Search?
Colin White, Arber Zela, Binxin Ru +2
Early methods in the rapidly developing field of neural architecture search (NAS) required fully training thousands of neural networks. To reduce this extreme computational cost, d…
Smooth Variational Graph Embeddings for Efficient Neural Architecture Search
Jovita Lukasik, David Friede, Arber Zela +2
Neural architecture search (NAS) has recently been addressed from various directions, including discrete, sampling-based methods and efficient differentiable approaches. While the…
Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks
Arber Zela, Julien Siems, Lucas Zimmer +3
The most significant barrier to the advancement of Neural Architecture Search (NAS) is its demand for large computational resources, which hinders scientifically sound empirical ev…
Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
Sheheryar Zaidi, Arber Zela, Thomas Elsken +3
Ensembles of neural networks achieve superior performance compared to stand-alone networks in terms of accuracy, uncertainty calibration and robustness to dataset shift. \emph{Deep…
NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
Arber Zela, Julien Siems, Frank Hutter
One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice. Nevertheless, there is still a lack of understanding…
Understanding and Robustifying Differentiable Architecture Search
Arber Zela, Thomas Elsken, Tonmoy Saikia +3
Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of…
AutoDispNet: Improving Disparity Estimation With AutoML
Tonmoy Saikia, Yassine Marrakchi, Arber Zela +2
Much research work in computer vision is being spent on optimizing existing network architectures to obtain a few more percentage points on benchmarks. Recent AutoML approaches pro…
Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
Arber Zela, Aaron Klein, Stefan Falkner +1
While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter…