2 citations · 3 across the 18 of their papers we have counts for
15 papers · 1 filter
Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion
Zeyu Liu, Jinhao Zhang, Yunquan Zhang +4
How can we determine whether a trained neural network is already deep enough? We study this under a fixed function-preserving residual-growth protocol specifying insertion location…
Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension
Jinhao Zhang, Zeyu Liu, Zicheng Yan +4
Existing theories of neural-network width characterize asymptotic limits, but provide limited guidance on whether an expansion direction identified from finite training data remain…
DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training
Yuanqing Wang, Yuchen Zhang, Hao Lin +9
Modern large language model (LLM) training is inherently dynamic: resource fluctuations, RLHF phase shifts, and cluster elasticity continually reshape the optimal parallelism layou…
A Qualitative Test-Risk Mechanism for Scaling Behavior in Normalized Residual Networks
Daning Cheng, Zeyu Liu, Jun Sun +4
The scaling behavior, in which test performance often improves as model size and data increase, is a central empirical phenomenon in modern deep learning, yet its theoretical basis…
MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
Xin Ye, Daning Cheng, Boyang Zhang +1
Training large-scale Mixture-of-Experts (MoE) models typically requires high-memory, high-bandwidth GPUs (e.g., A100), and their high cost has become a major barrier to large-model…
CALM: A CKA-Guided Adaptive Layer-Wise Modularization Framework for LLM Quantization
Jinhao Zhang, Yunquan Zhang, Daning Chen +2
Current mainstream post-training quantization methods for large language models typically apply a uniform quantization strategy across all network layers, overlooking the substanti…
HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioning
Jinhao Zhang, Yunquan Zhang, Zicheng yan +3
Post Training Quantization (PTQ), a mainstream model compression technique, often leads to the paradoxical 'low error, high loss' phenomenon because it focuses solely on minimizing…
Rethinking Parameter Sharing as Graph Coloring for Structured Compression
Boyang Zhang, Daning Cheng, Yunquan Zhang
Modern deep models have massive parameter sizes, leading to high inference-time memory usage that limits practical deployment. Parameter sharing, a form of structured compression,…
MoQE: Improve Quantization Model performance via Mixture of Quantization Experts
Jinhao Zhang, Yunquan Zhang, Boyang Zhang +2
Quantization method plays a crucial role in improving model efficiency and reducing deployment costs, enabling the widespread application of deep learning models on resource-constr…
Exploiting Block Coordinate Descent for Cost-Effective LLM Model Training
Zeyu Liu, Yan Li, Yunquan Zhang +6
Training large language models typically demands extensive GPU memory and substantial financial investment, which poses a barrier for many small- to medium-sized teams. In this pap…
A General Error-Theoretical Analysis Framework for Constructing Compression Strategies
Boyang Zhang, Daning Cheng, Yunquan Zhang +3
The exponential growth in parameter size and computational complexity of deep models poses significant challenges for efficient deployment. The core problem of existing compression…
Lossless Model Compression via Joint Low-Rank Factorization Optimization
Boyang Zhang, Daning Cheng, Yunquan Zhang +2
Low-rank factorization is a popular model compression technique that minimizes the error between approximated and original weight matrices. Despite achieving performances close…
FP=xINT:Representing Neural Networks via Low-Bit Series Basis Functions
Boyang Zhang, Daning Cheng, Yunquan Zhang +3
Post-Training Quantization (PTQ) converts pre-trained Full-Precision (FP) models into quantized versions without training. While existing methods reduce size and computational cost…
Asynch-SGBDT: Asynchronous Parallel Stochastic Gradient Boosting Decision Tree based on Parameters Server
Cheng Daning, Xia Fen, Li Shigang +1
In AI research and industry, machine learning is the most widely used tool. One of the most important machine learning algorithms is Gradient Boosting Decision Tree, i.e. GBDT whos…
Weighted parallel SGD for distributed unbalanced-workload training system
Cheng Daning, Li Shigang, Zhang Yunquan
Stochastic gradient descent (SGD) is a popular stochastic optimization method in machine learning. Traditional parallel SGD algorithms, e.g., SimuParallel SGD, often require all no…