6 papers
Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability
Ruixuan Huang, Yipei Wang, Wenyi Fang +7
Frontier large language model training consumes massive accelerator fleets and long wall-clock computation, making stability failures costly when they occur. After a numerical or a…
A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Models
Peifeng Gao, Wenyi Fang, Yang Zheng +1
Delayed loss spikes have been reported in neural-network training, but existing theory mainly explains earlier non-monotone behavior caused by overly large fixed learning rates. We…
Fast and Accurate Probing of In-Training LLMs' Downstream Performances
Zhichen Liu, Tianle Lun, Zhibin Wen +7
The paradigm of scaling Large Language Models (LLMs) in both parameter size and test time has pushed the boundaries of AI capabilities, but at the cost of making the traditional ge…
On the Spectral Flattening of Quantized Embeddings
Junlin Huang, Wenyi Fang, Zhenheng Tang +5
Training Large Language Models (LLMs) at ultra-low precision is critically impeded by instability rooted in the conflict between discrete quantization constraints and the intrinsic…
PLPP: Prompt Learning with Perplexity Is Self-Distillation for Vision-Language Models
Biao Liu, Wenyi Fang, Xiaoyu Wu +3
Pre-trained Vision-Language (VL) models such as CLIP have demonstrated their excellent performance across numerous downstream tasks. A recent method, Context Optimization (CoOp), f…
Training Overhead Ratio: A Practical Reliability Metric for Large Language Model Training Systems
Ning Lu, Qian Xie, Hao Zhang +4
Large Language Models (LLMs) are revolutionizing the AI industry with their superior capabilities. Training these models requires large-scale GPU clusters and significant computing…