4 papers
Sign-SGD via Parameter-Free Optimization
Daniil Medyakov, Sergey Stanko, Gleb Molodtsov +4
Large language models have achieved major advances across domains, yet training them remains extremely resource-intensive. We revisit Sign-SGD, which serves both as a memory-effici…
Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Sampling
Valery Parfenov, Grigoriy Evseev, Andrey Veprikov +3
Fine-tuning large pretrained language models (LLMs) is a cornerstone of modern NLP, yet its growing memory demands (driven by backpropagation and large optimizer States) limit depl…
Shuffling Heuristic in Variational Inequalities: Establishing New Convergence Guarantees
Daniil Medyakov, Gleb Molodtsov, Grigoriy Evseev +2
Variational inequalities have gained significant attention in machine learning and optimization research. While stochastic methods for solving these problems typically assume indep…
Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order
Egor Petrov, Grigoriy Evseev, Aleksey Antonov +4
Fine-tuning Large Language Models (LLMs) is essential for adapting pre-trained models to downstream tasks. Yet traditional first-order optimizers such as Stochastic Gradient Descen…