9 papers
MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models
Nurbek Tastan, Stefanos Laskaridis, Karthik Nandakumar +1
Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully. Hence, the trade-off b…
Response-Conditioned Parallel-to-Sequential Orchestration for Multi-Agent Systems
Nurbek Tastan, Alex Iacob, Lorenzo Sani +4
Multi-agent systems can solve complex tasks through collaboration between multiple Large Language Model agents. Existing collaboration frameworks typically operate in either a para…
Stochastic Self-Organization in Multi-Agent Systems
Nurbek Tastan, Samuel Horvath, Karthik Nandakumar
Multi-agent systems (MAS) based on Large Language Models (LLMs) have the potential to solve tasks that are beyond the reach of any single LLM. However, this potential can only be r…
LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuning
Nurbek Tastan, Stefanos Laskaridis, Martin Takac +2
Large pre-trained models are commonly adapted to downstream tasks using parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA), which injects small trainable lo…
CYCle: Choosing Your Collaborators Wisely to Enhance Collaborative Fairness in Decentralized Learning
Nurbek Tastan, Samuel Horvath, Karthik Nandakumar
Collaborative learning (CL) enables multiple participants to jointly train machine learning (ML) models on decentralized data sources without raw data sharing. While the primary go…
Aequa: Fair Model Rewards in Collaborative Learning via Slimmable Networks
Nurbek Tastan, Samuel Horvath, Karthik Nandakumar
Collaborative learning enables multiple participants to learn a single global model by exchanging focused updates instead of sharing data. One of the core challenges in collaborati…