5 papers · 1 filter
Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search
Zhiliang Chen, Sebastian Ament, David Eriksson +4
Optimal hyperparameter scaling laws describe how the best hyperparameters for large language model (LLM) training change with model and data scale, enabling practitioners to predic…
BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks
Ruth Wan Theng Chew, Zhiliang Chen, Apivich Hemachandra +1
Optimization of LLM training and inference configurations, such as hyperparameters, data mixtures, and prompts, is critical to performance, but it is often approached heuristically…
The Chicken and Egg Dilemma: Co-optimizing Data and Model Configurations for LLMs
Zhiliang Chen, Alfred Wei Lun Leong, Shao Yong Ong +6
Co-optimizing data and model configurations for training LLMs presents a classic chicken-and-egg dilemma: The best training data configuration (e.g., data mixture) for a downstream…
Uncovering Scaling Laws for Large Language Models via Inverse Problems
Arun Verma, Zhaoxuan Wu, Zijian Zhou +15
Large Language Models (LLMs) are large-scale pretrained models that have achieved remarkable success across diverse domains. These successes have been driven by unprecedented compl…
DUET: Optimizing Training Data Mixtures via Feedback from Unseen Evaluation Tasks
Zhiliang Chen, Gregory Kang Ruey Lau, Chuan-Sheng Foo +1
The performance of an LLM depends heavily on the relevance of its training data to the downstream evaluation task. However, in practice, the data involved in an unseen evaluation t…