2 citations · 3 across the 8 of their papers we have counts for
8 papers
Less is MoE: Trimming Experts in Domain-Specialist Language Models
Haoze He, Xinkai Zou, Xuan Jiang +4
Mixture-of-Experts (MoE) models achieve strong performance through conditional computation, but their large parameter footprint poses deployment challenges. Prior MoE compression a…
Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
Haoze He, Xingyuan Ding, Xuan Jiang +5
Despite MoE models leading many benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layers are fragile. Methods such as DenseMix…
RAST-MoE-RL: A Regime-Aware Spatio-Temporal MoE Framework for Deep Reinforcement Learning in Ride-Hailing
Yuhan Tang, Kangxin Cui, Jung Ho Park +6
Ride-hailing platforms face the challenge of balancing passenger waiting times with overall system efficiency under highly uncertain supply-demand conditions. Adaptive delayed matc…
Towards Generalizable Context-aware Anomaly Detection: A Large-scale Benchmark in Cloud Environments
Xinkai Zou, Xuan Jiang, Ruikai Huang +8
Anomaly detection in cloud environments remains both critical and challenging. Existing context-level benchmarks typically focus on either metrics or logs and often lack reliable a…
Sparse Matrix in Large Language Model Fine-tuning
Haoze He, Juncheng Billy Li, Xuan Jiang +1
LoRA and its variants have become popular parameter-efficient fine-tuning (PEFT) methods due to their ability to avoid excessive computational costs. However, an accuracy gap often…
Adjacent Leader Decentralized Stochastic Gradient Descent
Haoze He, Jing Wang, Anna Choromanska
This work focuses on the decentralized deep learning optimization framework. We propose Adjacent Leader Decentralized Gradient Descent (AL-DSGD), for improving final model performa…
Accelerating Parallel Stochastic Gradient Descent via Non-blocking Mini-batches
Haoze He, Parijat Dube
SOTA decentralized SGD algorithms can overcome the bandwidth bottleneck at the parameter server by using communication collectives like Ring All-Reduce for synchronization. While t…
RCD-SGD: Resource-Constrained Distributed SGD in Heterogeneous Environment via Submodular Partitioning
Haoze He, Parijat Dube
The convergence of SGD based distributed training algorithms is tied to the data distribution across workers. Standard partitioning techniques try to achieve equal-sized partitions…