19 citations · 58 across the 13 of their papers we have counts for
14 papers
ScAn-Bench: Evaluating Scaling Analysis Methodology
Artin Sermaxhaj, Nastaran Alipour, Donat Sinani +5
Recent progress in machine learning is driven by large-scale foundation models, where scaling laws and finding optimal scaling prescriptions for architecture, data, and hyperparame…
Amortizing Scaling Law Construction Costs
Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6
Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and paramete…
When is Warmstarting Effective for Scaling Language Models?
Neeratyoy Mallik, Maciej Janowski, Johannes Hog +4
Model growth from a given checkpoint aims to accelerate training of a larger model, offering potential resource savings. Despite recent interest, warmstarting has seen limited prac…
carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks
Carolin Benjamins, Helena Graf, Sarah Segel +14
Hyperparameter Optimization (HPO) is crucial to develop well-performing machine learning models. In order to ease prototyping and benchmarking of HPO methods, we propose carps, a b…
Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization
Timur Carstensen, Neeratyoy Mallik, Frank Hutter +1
As model sizes grow, finding efficient and cost-effective hyperparameter optimization (HPO) methods becomes increasingly crucial for deep learning pipelines. While multi-fidelity H…
Warmstarting for Scaling Language Models
Neeratyoy Mallik, Maciej Janowski, Johannes Hog +4
Scaling model sizes to scale performance has worked remarkably well for the current large language models paradigm. The research and empirical findings of various scaling studies l…
LMEMs for post-hoc analysis of HPO Benchmarking
Anton Geburek, Neeratyoy Mallik, Danny Stoll +2
The importance of tuning hyperparameters in Machine Learning (ML) and Deep Learning (DL) is established through empirical research and applications, evident from the increase in ne…
In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization
Herilalaina Rakotoarison, Steven Adriaensen, Neeratyoy Mallik +3
With the increasing computational costs associated with deep learning, automated hyperparameter optimization methods, strongly relying on black-box Bayesian optimization (BO), face…
Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarks
Shuhei Watanabe, Neeratyoy Mallik, Edward Bergman +1
While deep learning has celebrated many successes, its results often hinge on the meticulous selection of hyperparameters (HPs). However, the time-consuming nature of deep learning…
PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning
Neeratyoy Mallik, Edward Bergman, Carl Hvarfner +5
Hyperparameters of Deep Learning (DL) pipelines are crucial for their downstream performance. While a large number of methods for Hyperparameter Optimization (HPO) have been develo…
HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik +6
To achieve peak predictive performance, hyperparameter optimization (HPO) is a crucial component of machine learning and its applications. Over the last years, the number of effici…
Squirrel: A Switching Hyperparameter Optimizer
Noor Awad, Gresa Shala, Difan Deng +9
In this short note, we describe our submission to the NeurIPS 2020 BBO challenge. Motivated by the fact that different optimizers work well on different problems, our approach swit…
Differential Evolution for Neural Architecture Search
Noor Awad, Neeratyoy Mallik, Frank Hutter
Neural architecture search (NAS) methods rely on a search strategy for deciding which architectures to evaluate next and a performance estimation strategy for assessing their perfo…
OpenML-Python: an extensible Python API for OpenML
Matthias Feurer, Jan N. van Rijn, Arlind Kadra +6
OpenML is an online platform for open science collaboration in machine learning, used to share datasets and results of machine learning experiments. In this paper we introduce Open…