5 papers
Understanding Stability of Outer Nesterov Momentum on Stochastic Pseudo-Gradients
Nishant Sharma, Laura Gomezjurado Gonzalez, Ganesh Talluri +2
Large neural networks are commonly trained across many workers, and communication between them is a major cost. Workers therefore take several local optimization steps before sendi…
Introspecting Alignment Shifts Beyond Behaviors Implanted Through Fine-Tuning
Kotaro Yoshida, Laura Gomezjurado Gonzalez, Yukinori Yamamoto +3
Fine-tuning enables a source model to acquire desired capabilities and behaviors in a target domain while retaining much of its general-purpose competence. However, this adaptation…
Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization
Tatsuhiro Nakamori, Laura Gomezjurado Gonzalez, Ganesh Talluri +5
Low-rank gradient compression reduces communication in distributed training by representing updates with rank- factors. Dion is a recent method that approximates Muon, a spectra…
The Long Delay to Arithmetic Generalization
Laura Gomezjurado Gonzalez
Large language models today still make mistakes on multi-digit arithmetic when they do not have tools, and part of the reason is how numbers are split into tokens. Arithmetic is al…
On Fairness of Task Arithmetic: The Role of Task Vectors
Hiroki Naganuma, Kotaro Yoshida, Laura Gomezjurado Gonzalez +3
Model editing techniques, particularly task arithmetic with task vectors, offer an efficient alternative to full fine-tuning by enabling direct parameter updates through simple ari…