5 papers
CORTEX: Learning to Share and Specialize in Dense Language Models
Chuiyang Meng, Ming Tang, Vincent W. S. Wong
Large language models are trained on heterogeneous data mixtures, where different knowledge domains require both shared knowledge and specialization. Existing modular approaches ty…
Social Circuits behind Multi-agent Echo Chambers
Chuiyang Meng, Wenlu Yu, Ming Tang +1
Language-model agents exchange messages to combine evidence, but their communication can also create echo chambers that reinforce shared errors. However, overall task performance d…
FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment
Chuiyang Meng, Ming Tang, Vincent W. S. Wong
Parameter-efficient fine-tuning techniques such as low-rank adaptation (LoRA) enable large language models (LLMs) to adapt to downstream tasks efficiently. Federated learning (FL)…
ASFL: An Adaptive Model Splitting and Resource Allocation Framework for Split Federated Learning
Chuiyang Meng, Ming Tang, Vincent W. S. Wong
Federated learning (FL) enables multiple clients to collaboratively train a machine learning model without sharing their raw data. However, the limited computation resources of the…
ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation
Chuiyang Meng, Ming Tang, Vincent W. S. Wong
Federated fine-tuning of large language models (LLMs) enables collaborative tuning across distributed clients. However, due to the large size of LLMs, local updates in federated le…