FedMS: Federated Learning with Mixture of Sparsely Activated Foundations Models

Panlong Wu, Kangshuo Li, Ting Wang +1

Foundation models have shown great success in natural language processing, computer vision, and multimodal tasks. FMs have a large number of model parameters, thus requiring a subs…