Showing 2025Show all
3 papers · 1 filter
cs.CL2025
SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs
Ruixuan Huang, Hao Zeng, Hantao Huang +4
Post-training quantization (PTQ) plays a crucial role in the democratization of large language models (LLMs). However, existing low-bit quantization and sparsification techniques a…
cs.CR2025
RepetitionCurse: Measuring and Understanding Router Imbalance in Mixture-of-Experts LLMs under DoS Stress
Ruixuan Huang, Qingyue Wang, Hantao Huang +4
Mixture-of-Experts architectures have become the standard for scaling large language models due to their superior parameter efficiency. To accommodate the growing number of experts…
cs.LG2025
Scaling Law Analysis in Federated Learning: How to Select the Optimal Model Size?
Xuanyu Chen, Nan Yang, Shuai Wang +1
The recent success of large language models (LLMs) has sparked a growing interest in training large-scale models. As the model size continues to scale, concerns are growing about t…