11 citations · 19 across the 16 of their papers we have counts for
17 papers
Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrix
Fangshuo Liao, Anastasios Kyrillidis
The phenomenon of the top subspace stabilization of the Hessian matrix is an surprising and critical aspect in study of the second-order information of neural network training. Pri…
AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery
Barbara Su, Fangshuo Liao, Anastasios Kyrillidis
Fine-tuning large language models with LoRA requires choosing a rank r before training starts. Existing approaches either extract rank-1 components sequentially, freezing each comp…
SGD at the Edge of Stability: The Stochastic Sharpness Gap
Fangshuo Liao, Afroditi Kolomvaki, Anastasios Kyrillidis
When training neural networks with full-batch gradient descent (GD) and step size , the largest eigenvalue of the Hessian -- the sharpness -- rises to an…
Convergence Analysis of Two-Layer Neural Networks under Gaussian Input Masking
Afroditi Kolomvaki, Fangshuo Liao, Evan Dramko +2
We investigate the convergence guarantee of two-layer neural network training with Gaussian randomly masked inputs. This scenario corresponds to Gaussian dropout at the input level…
Exploiting Low-Rank Objective Structure in Discrete Quadratic Optimization
Ria Stevens, Fangshuo Liao, Barbara Su +3
We study the problem of maximizing a complex-valued quadratic form over the roots of unity. We show that when the objective matrix $\mathbf{Q}^\star \in \mathbb{C}^…
Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts
Fangshuo Liao, Anastasios Kyrillidis
Mixture-of-Experts (MoE) architectures have emerged as a cornerstone of modern AI systems. In particular, MoEs route inputs dynamically to specialized experts whose outputs are agg…
One Rank at a Time: Cascading Error Dynamics in Sequential Learning
Mahtab Alizadeh Vandchali, Fangshuo, Liao +1
Sequential learning -- where complex tasks are broken down into simpler, hierarchical components -- has emerged as a paradigm in AI. This paper views sequential learning through th…
Provable Model-Parallel Distributed Principal Component Analysis with Parallel Deflation
Fangshuo Liao, Wenyi Su, Anastasios Kyrillidis
We study a distributed Principal Component Analysis (PCA) framework where each worker targets a distinct eigenvector and refines its solution by updating from intermediate solution…
On the Error-Propagation of Inexact Hotelling's Deflation for Principal Component Analysis
Fangshuo Liao, Junhyung Lyle Kim, Cruz Barnum +1
Principal Component Analysis (PCA) aims to find subspaces spanned by the so-called principal components that best represent the variance in the dataset. The deflation method is a p…
Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings
Yehya Farhat, Hamza ElMokhtar Shili, Fangshuo Liao +7
Mixture-of-Experts (MoEs) achieve scalability by dynamically activating subsets of their components. Yet, understanding how expertise emerges through joint training of gating mecha…
Provable Accelerated Convergence of Nesterov's Momentum for Deep ReLU Neural Networks
Fangshuo Liao, Anastasios Kyrillidis
Current state-of-the-art analyses on the convergence of gradient descent for training neural networks focus on characterizing properties of the loss landscape, such as the Polyak-L…
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
Zichang Liu, Aditya Desai, Fangshuo Liao +5
Large language models(LLMs) have sparked a new wave of exciting AI applications. Hosting these models at scale requires significant memory resources. One crucial memory bottleneck…
Strong Lottery Ticket Hypothesis with --perturbation
Zheyang Xiong, Fangshuo Liao, Anastasios Kyrillidis
The strong Lottery Ticket Hypothesis (LTH) claims the existence of a subnetwork in a sufficiently large, randomly initialized neural network that approximates some target neural ne…
LOFT: Finding Lottery Tickets through Filter-wise Training
Qihan Wang, Chen Dun, Fangshuo Liao +2
Recent work on the Lottery Ticket Hypothesis (LTH) shows that there exist ``\textit{winning tickets}'' in large neural networks. These tickets represent ``sparse'' versions of the…
On the Convergence of Shallow Neural Network Training with Randomly Masked Neurons
Fangshuo Liao, Anastasios Kyrillidis
With the motive of training all the parameters of a neural network, we study why and when one can achieve this by iteratively creating, training, and combining randomly selected su…
How much pre-training is enough to discover a good subnetwork?
Cameron R. Wolfe, Fangshuo Liao, Qihan Wang +2
Neural network pruning is useful for discovering efficient, high-performing subnetworks within pre-trained, dense network architectures. More often than not, it involves a three-st…
GIST: Distributed Training for Large-Scale Graph Convolutional Networks
Cameron R. Wolfe, Jingkang Yang, Arindam Chowdhury +4
The graph convolutional network (GCN) is a go-to solution for machine learning on graphs, but its training is notoriously difficult to scale both in terms of graph size and the num…