4 papers · 1 filter
Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs
Afsara Benazir, Chen Chen, Rongxiao Qu +3
Mixture-of-Experts (MoE) LLMs scale model capacity efficiently through sparse activation, but their large expert parameter footprint, routing imbalance, and long-context KV-cache g…
StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold
Zhizhong Li, Sina Sajadmanesh, Jingtao Li +1
Low-rank adaptation (LoRA) has been widely adopted as a parameter-efficient technique for fine-tuning large-scale pre-trained models. However, it still lags behind full fine-tuning…
FEDEXCHANGE: Bridging the Domain Gap in Federated Object Detection for Free
Haolin Yuan, Jingtao Li, Weiming Zhuang +2
Federated Object Detection (FOD) enables clients to collaboratively train a global object detection model without accessing their local data from diverse domains. However, signific…
Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptation
Guangyu Sun, Jingtao Li, Weiming Zhuang +2
Foundation models (FMs) exhibit remarkable generalization but require adaptation to downstream tasks, particularly in privacy-sensitive applications. Due to data privacy regulation…