12 papers
Predicting LLM Output Length via Entropy-Guided Representations
Huanyi Xie, Yubin Chen, Liangyu Wang +2
The long-tailed distribution of sequence lengths in LLM serving and reinforcement learning (RL) sampling causes significant computational waste due to excessive padding in batched…
Differentially Private Non-convex Distributionally Robust Optimization
Difei Xu, Meng Ding, Zebin Ma +4
Real-world deployments routinely face distribution shifts, group imbalances, and adversarial perturbations, under which the traditional Empirical Risk Minimization (ERM) framework…
Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Data
Jiaming Zhang, Huanyi Xie, Meng Ding +3
Recent research shows that modern deep learning models achieve high predictive accuracy partly by memorizing individual training samples. Such memorization raises serious privacy c…
PAHQ: Accelerating Automated Circuit Discovery through Mixed-Precision Inference Optimization
Xinhai Wang, Shu Yang, Liangyu Wang +4
Circuit discovery, which involves identifying sparse and task-relevant subnetworks in pre-trained language models, is a cornerstone of mechanistic interpretability. Automated Circu…
Attributing Data for Sharpness-Aware Minimization
Chenyang Ren, Yifan Jia, Huanyi Xie +5
Sharpness-aware Minimization (SAM) improves generalization in large-scale model training by linking loss landscape geometry to generalization. However, challenges such as mislabele…
DistZO2: High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computing
Liangyu Wang, Huanyi Xie, Di Wang
Fine-tuning large language models (LLMs) remains resource-intensive due to their sheer scale. While zeroth-order (ZO) optimization provides a memory-efficient alternative by elimin…