354 citations · 1.4k across the 34 of their papers we have counts for
3 papers · 1 filter
SMILE: Scaling Mixture-of-Experts with Efficient Bi-level Routing
Chaoyang He, Shuai Zheng, Aston Zhang +4
The mixture of Expert (MoE) parallelism is a recent advancement that scales up the model size with constant computational cost. MoE selects different sets of parameters (i.e., expe…
FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings
Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers +21
Federated Learning (FL) is a novel approach enabling several clients holding sensitive data to collaboratively train machine learning models, without centralizing data. The cross-s…
Partial Model Averaging in Federated Learning: Performance Guarantees and Benefits
Sunwoo Lee, Anit Kumar Sahu, Chaoyang He +1
Local Stochastic Gradient Descent (SGD) with periodic model averaging (FedAvg) is a foundational algorithm in Federated Learning. The algorithm independently runs SGD on multiple w…