4 papers
Spectral Scaling Laws of Muon
Gagik Magakyan, Pablo Parrilo, Asuman Ozdaglar
Orthonormalized update rules have rapidly become a leading choice of optimizer for training large language models, with recent open-source state-of-the-art models adopting Muon. To…
Collaborative and Efficient Fine-tuning: Leveraging Task Similarity
Gagik Magakyan, Amirhossein Reisizadeh, Chanwoo Park +2
Adaptability has been regarded as a central feature in the foundation models, enabling them to effectively acclimate to unseen downstream tasks. Parameter-efficient fine-tuning met…
Dion: Distributed Orthonormalized Updates
Kwangjun Ahn, Byron Xu, Natalie Abreu +5
Orthonormalized updates accelerate training, improve stability, and enable robust hyperparameter transfer, but existing methods like Muon rely on dense matrix operations that clash…
General framework for online-to-nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimization
Kwangjun Ahn, Gagik Magakyan, Ashok Cutkosky
This work investigates the effectiveness of schedule-free methods, developed by A. Defazio et al. (NeurIPS 2024), in nonconvex optimization settings, inspired by their remarkable e…