4 papers
A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Models
Peifeng Gao, Wenyi Fang, Yang Zheng +1
Delayed loss spikes have been reported in neural-network training, but existing theory mainly explains earlier non-monotone behavior caused by overly large fixed learning rates. We…
Fast and Accurate Probing of In-Training LLMs' Downstream Performances
Zhichen Liu, Tianle Lun, Zhibin Wen +7
The paradigm of scaling Large Language Models (LLMs) in both parameter size and test time has pushed the boundaries of AI capabilities, but at the cost of making the traditional ge…
On the Spectral Flattening of Quantized Embeddings
Junlin Huang, Wenyi Fang, Zhenheng Tang +5
Training Large Language Models (LLMs) at ultra-low precision is critically impeded by instability rooted in the conflict between discrete quantization constraints and the intrinsic…
PLPP: Prompt Learning with Perplexity Is Self-Distillation for Vision-Language Models
Biao Liu, Wenyi Fang, Xiaoyu Wu +3
Pre-trained Vision-Language (VL) models such as CLIP have demonstrated their excellent performance across numerous downstream tasks. A recent method, Context Optimization (CoOp), f…