Showing cs.LGShow all
3 papers · 1 filter
cs.LG2026
Clean: Second-order LLM Training at Linear Memory Cost via Nystr{ö}m Sketching
Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi Shikai Fang +4
Training large language models (LLMs) entails a fundamental trade-off: memory-efficient optimizers such as Adam discard cross-parameter curvature, whereas full-curvature methods su…
cs.LG2025
Geometrically Principled Randomized Optimization for Efficient LLM Training
Sahar Rajabi, Nayeema Nonta, Sirisha Rambhatla
Low-rank gradient optimization for large language models is currently divided into two categories: structured methods that rigorously identify subspaces, and randomized approaches…
cs.LG2025
SubTrack++ : Gradient Subspace Tracking for Scalable LLM Training
Sahar Rajabi, Nayeema Nonta, Sirisha Rambhatla
Training large language models (LLMs) is highly resource-intensive due to their massive number of parameters and the overhead of optimizer states. While recent work has aimed to re…