BSFA: Leveraging the Subspace Dichotomy to Accelerate Neural Network Training

Wenjie Zhou, Bohan Wang, Wei Chen +1

Recent studies \citep{gur2018gradient,song2024does, wen2024understanding} highlight a fundamental dichotomy in deep learning optimization: Although parameter updates along the top…