9 papers · 1 filter
A Qualitative Test-Risk Mechanism for Scaling Behavior in Normalized Residual Networks
Daning Cheng, Zeyu Liu, Jun Sun +4
The scaling behavior, in which test performance often improves as model size and data increase, is a central empirical phenomenon in modern deep learning, yet its theoretical basis…
MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
Xin Ye, Daning Cheng, Boyang Zhang +1
Training large-scale Mixture-of-Experts (MoE) models typically requires high-memory, high-bandwidth GPUs (e.g., A100), and their high cost has become a major barrier to large-model…
HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioning
Jinhao Zhang, Yunquan Zhang, Zicheng yan +3
Post Training Quantization (PTQ), a mainstream model compression technique, often leads to the paradoxical 'low error, high loss' phenomenon because it focuses solely on minimizing…
Rethinking Parameter Sharing as Graph Coloring for Structured Compression
Boyang Zhang, Daning Cheng, Yunquan Zhang
Modern deep models have massive parameter sizes, leading to high inference-time memory usage that limits practical deployment. Parameter sharing, a form of structured compression,…
MoQE: Improve Quantization Model performance via Mixture of Quantization Experts
Jinhao Zhang, Yunquan Zhang, Boyang Zhang +2
Quantization method plays a crucial role in improving model efficiency and reducing deployment costs, enabling the widespread application of deep learning models on resource-constr…
Exploiting Block Coordinate Descent for Cost-Effective LLM Model Training
Zeyu Liu, Yan Li, Yunquan Zhang +6
Training large language models typically demands extensive GPU memory and substantial financial investment, which poses a barrier for many small- to medium-sized teams. In this pap…
A General Error-Theoretical Analysis Framework for Constructing Compression Strategies
Boyang Zhang, Daning Cheng, Yunquan Zhang +3
The exponential growth in parameter size and computational complexity of deep models poses significant challenges for efficient deployment. The core problem of existing compression…
Lossless Model Compression via Joint Low-Rank Factorization Optimization
Boyang Zhang, Daning Cheng, Yunquan Zhang +2
Low-rank factorization is a popular model compression technique that minimizes the error between approximated and original weight matrices. Despite achieving performances close…
FP=xINT:Representing Neural Networks via Low-Bit Series Basis Functions
Boyang Zhang, Daning Cheng, Yunquan Zhang +3
Post-Training Quantization (PTQ) converts pre-trained Full-Precision (FP) models into quantized versions without training. While existing methods reduce size and computational cost…