From Task Mixtures to Specialized Experts

Hojat Allah Salehi, Mehrdad Mahdavi, Andrew Arash Mahyari +1

In collaborative foundation model fine-tuning, client data is rarely homogeneous. Instead, clients typically possess unknown mixtures of distinct data distributions, or tasks. Conv…