3 papers
cs.LG2026
RQ-MoE: Residual Quantization via Mixture of Experts for Efficient Input-Dependent Vector Compression
Zhengjia Zhong, Shuyan Ke, Zaizhou Lin +3
Vector quantization is a fundamental tool for compressing high-dimensional embeddings, yet existing multi-codebook methods rely on static codebooks that limit expressiveness under…
cs.LG2026
DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism
Zhichen Zeng, Chi-Chih Chang, Jiayi Wang +10
Mixture-of-experts (MoE) architectures enable trillion-parameter LLMs with sparsely activated experts. Expert parallelism (EP) is a widely adopted MoE training strategy, but it suf…
cs.DC2026
veScale-FSDP: Flexible and High-Performance FSDP at Scale
Zezhou Wang, Youjie Li, Zhiqi Lin +9
Fully Sharded Data Parallel (FSDP), also known as Zero Redundancy Optimizer (ZeRO), is widely used for large-scale model training, because of its memory efficiency and minimal intr…