2 citations · 4 across the 24 of their papers we have counts for
9 papers · 1 filter
Building a Nested LADDER for LLM Unstructured Post-Training Pruning
Nicolás Mauricio Cuadrado Ávila, Ivo Gollini Navarrete, Martin Takáč +1
Non-uniform allocation can improve post-training pruning of large language models, yet quality can still degrade sharply at high sparsity. To control degradation as sparsity increa…
LionMuon: Alternating Spectral and Sign Descent for Efficient Training
Arman Bolatov, Artem Riabinin, Nikita Kornilov +4
Pretraining a language model takes enormous compute, and the right optimizer can save a good part of it. Muon's spectral step gives a stronger direction than a sign step, but it is…
Optimizers for Diffusion Models: A Controlled Benchmark
Arman Bolatov, Egor Shulgin, David Li +6
Discrete diffusion models now match autoregressive language models on several benchmarks, while the question of how best to train them has received far less attention: the optimize…
CacheMuon: Using Temporal Preconditioning To Approximate Polar Factor
Bishnu Dev, Sushil Bohara, Martin Takáč +1
Muon is an optimizer that computes updates using the polar factor of the momentum matrix and has shown strong empirical performance across a range of training settings. A key compo…
Value-Gradient Hypothesis of RL for LLMs
Arip Asadulaev, Daniil Ognev, Karim Salta +1
Reinforcement learning substantially improves pretrained language models, but it remains understudied why critic-free methods such as PPO and GRPO work as well as they do, and when…
LionMuon: Alternating Spectral and Sign Descent for Efficient Training
Arman Bolatov, Artem Riabinin, Nikita Kornilov +4
In large-scale optimization, the cheapness and effectiveness of update steps are the most crucial factors for a successful optimizer. Sign-based optimizers like Lion or Signum prod…
Byzantine-Robust Optimization under -Smoothness
Arman Bolatov, Samuel Horváth, Martin Takáč +1
We consider distributed optimization under Byzantine attacks in the presence of -smoothness, a generalization of standard -smoothness that captures functions with sta…
Breaking the Martingale Curse: Multi-Agent Debate via Asymmetric Cognitive Potential Energy
Yuhan Liu, Juntian Zhang, Yichen Wu +4
Multi-Agent Debate (MAD) has emerged as a promising paradigm for enhancing large language model reasoning. However, recent work reveals a limitation:standard MAD cannot improve bel…
Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC
Abdulla Jasem Almansoori, Maria Ivanova, Andrey Veprikov +3
Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights, dramatically reducing trainable parameters and memory. In this work, we ad…