OASIS: Online Activation Subspace Learning for Memory-Efficient Training

Sakshi Choudhary, Utkarsh Saxena, Kaushik Roy

Training large language models (LLMs) is constrained by memory requirements, with activations accounting for a substantial fraction of the total footprint. Existing approaches redu…