Multi-armed bandits for resource efficient, online optimization of language model pre-training: the use case of dynamic masking

Iñigo Urteaga, Moulay-Zaïdane Draïdia, Tomer Lancewicki +1

We design and evaluate a Bayesian optimization framework for resource efficient pre-training of Transformer-based language models (TLMs). TLM pre-training requires high computation…