123 citations · 218 across the 19 of their papers we have counts for
3 papers · 1 filter
Efficient Long-Horizon Learning for Learned Optimization
Xiaolong Huang, Benjamin Thérien, James Harrison +1
Learned optimization aims to improve upon hand-designed optimizers (e.g., Adam and Muon) by meta-learning small neural network optimizers over a distribution of tasks. While recent…
PyLO: Towards Accessible Learned Optimizers in PyTorch
Paul Janson, Benjamin Therien, Quentin Anthony +3
Learned optimizers have been an active research topic over the past decade, with increasing progress toward practical, general-purpose optimizers that can serve as drop-in replacem…
MuLoCo: Muon is a practical inner optimizer for DiLoCo
Benjamin Thérien, Xiaolong Huang, Aaron Defazio +2
DiLoCo is a powerful framework for training large language models (LLMs), enabling larger optimal batch sizes and increased accelerator utilization under networking constraints. Ho…