12 citations · 53 across the 37 of their papers we have counts for
6 papers · 2 filters
Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrix
Fangshuo Liao, Anastasios Kyrillidis
The phenomenon of the top subspace stabilization of the Hessian matrix is an surprising and critical aspect in study of the second-order information of neural network training. Pri…
SuperThoughts: Reasoning Tokens in Superposition
Zheyang Xiong, Shivam Garg, Max Yu +4
Long Chain-of-Thought (CoT) reasoning improves LLM problem-solving but is computationally expensive due to sequential token generation. While recent works explore reasoning in cont…
One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer
Jucheng Shen, Barbara Su, Anastasios Kyrillidis
Can a shared-weight recurrent Transformer develop distinct internal roles without being partitioned into separate modules? We study this in Asymmetric Input Recurrence (AIR), a min…
AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery
Barbara Su, Fangshuo Liao, Anastasios Kyrillidis
Fine-tuning large language models with LoRA requires choosing a rank r before training starts. Existing approaches either extract rank-1 components sequentially, freezing each comp…
SGD at the Edge of Stability: The Stochastic Sharpness Gap
Fangshuo Liao, Afroditi Kolomvaki, Anastasios Kyrillidis
When training neural networks with full-batch gradient descent (GD) and step size , the largest eigenvalue of the Hessian -- the sharpness -- rises to an…
Convergence Analysis of Two-Layer Neural Networks under Gaussian Input Masking
Afroditi Kolomvaki, Fangshuo Liao, Evan Dramko +2
We investigate the convergence guarantee of two-layer neural network training with Gaussian randomly masked inputs. This scenario corresponds to Gaussian dropout at the input level…