25 citations · 39 across the 12 of their papers we have counts for
6 papers · 1 filter
Swimba: Switch Mamba Model Scales State Space Models
Zhixu Du, Krishna Teja Chitty-Venkata, Murali Emani +3
Mixture-of-experts (MoE) is a common approach for increasing parameter capacity, but applying MoE to state space model (SSM) token mixers can multiply the cost of the recurrent sta…
FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models
Zishan Shao, Yixiao Wang, Qinsi Wang +6
Singular Value Decomposition (SVD) has recently seen a surge of interest as a simple yet powerful tool for large language models (LLMs) compression, with a growing number of works…
MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing
Haiyue Ma, Zhixu Du, Yiran Chen
In multi-GPU Mixture-of-Experts (MoE) network, experts are distributed across different GPUs, which creates load imbalance as each expert processes different number of tokens. Rece…
SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models
Zhixu Du, Shiyu Li, Yuhao Wu +7
Mixture-of-Experts (MoE) has emerged as a favorable architecture in the era of large models due to its inherent advantage, i.e., enlarging model capacity without incurring notable…
Rethinking Normalization Methods in Federated Learning
Zhixu Du, Jingwei Sun, Ang Li +4
Federated learning (FL) is a popular distributed learning framework that can reduce privacy risks by not explicitly sharing private data. In this work, we explicitly uncover extern…
Improved Input Reprogramming for GAN Conditioning
Tuan Dinh, Daewon Seo, Zhixu Du +2
We study the GAN conditioning problem, whose goal is to convert a pretrained unconditional GAN into a conditional GAN using labeled data. We first identify and analyze three approa…