2 citations · 3 across the 8 of their papers we have counts for
6 papers · 1 filter
Less is MoE: Trimming Experts in Domain-Specialist Language Models
Haoze He, Xinkai Zou, Xuan Jiang +4
Mixture-of-Experts (MoE) models achieve strong performance through conditional computation, but their large parameter footprint poses deployment challenges. Prior MoE compression a…
Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
Haoze He, Xingyuan Ding, Xuan Jiang +5
Despite MoE models leading many benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layers are fragile. Methods such as DenseMix…
RAST-MoE-RL: A Regime-Aware Spatio-Temporal MoE Framework for Deep Reinforcement Learning in Ride-Hailing
Yuhan Tang, Kangxin Cui, Jung Ho Park +6
Ride-hailing platforms face the challenge of balancing passenger waiting times with overall system efficiency under highly uncertain supply-demand conditions. Adaptive delayed matc…
Adjacent Leader Decentralized Stochastic Gradient Descent
Haoze He, Jing Wang, Anna Choromanska
This work focuses on the decentralized deep learning optimization framework. We propose Adjacent Leader Decentralized Gradient Descent (AL-DSGD), for improving final model performa…
Accelerating Parallel Stochastic Gradient Descent via Non-blocking Mini-batches
Haoze He, Parijat Dube
SOTA decentralized SGD algorithms can overcome the bandwidth bottleneck at the parameter server by using communication collectives like Ring All-Reduce for synchronization. While t…
RCD-SGD: Resource-Constrained Distributed SGD in Heterogeneous Environment via Submodular Partitioning
Haoze He, Parijat Dube
The convergence of SGD based distributed training algorithms is tied to the data distribution across workers. Standard partitioning techniques try to achieve equal-sized partitions…