OPTScientist: Multi-Agent Discovery of Typed Optimizer Programs for Transformer Pretraining

Zhongzheng Li, Tiancan Feng, Wenhao Li +5

Designing optimizers for modern deep learning remains a challenging scientific problem, requiring the joint consideration of optimization geometry, state dynamics, numerical stabil…