2 papers
cs.LG2025
Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
Lior Belenki, Alekh Agarwal, Tianze Shi +1
We propose a method to optimize language model pre-training data mixtures through efficient approximation of the cross-entropy loss corresponding to each candidate mixture via a Mi…
cs.LG2024
The Vizier Gaussian Process Bandit Algorithm
Xingyou Song, Qiuyi Zhang, Chansoo Lee +8
Google Vizier has performed millions of optimizations and accelerated numerous research and production systems at Google, demonstrating the success of Bayesian optimization as a la…