178 citations · 549 across the 48 of their papers we have counts for
8 papers · 1 filter
Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
Dujian Ding, Ankur Mallick, Chi Wang +5
Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size, while smaller models that can be deployed on lower c…
Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings
Yehya Farhat, Hamza ElMokhtar Shili, Fangshuo Liao +7
Mixture-of-Experts (MoEs) achieve scalability by dynamically activating subsets of their components. Yet, understanding how expertise emerges through joint training of gating mecha…
ADMoE: Anomaly Detection with Mixture-of-Experts from Noisy Labels
Yue Zhao, Guoqing Zheng, Subhabrata Mukherjee +2
Existing works on anomaly detection (AD) rely on clean labels from human annotators that are expensive to acquire in practice. In this work, we propose a method to leverage weak/no…
Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners
Shashank Gupta, Subhabrata Mukherjee, Krishan Subudhi +4
Traditional multi-task learning (MTL) methods use dense networks that use the same set of shared weights across several different tasks. This often creates interference where two o…
The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy
Tianlong Chen, Zhenyu Zhang, Yu Cheng +2
Vision transformers (ViTs) have gained increasing popularity as they are commonly believed to own higher modeling capacity and representation flexibility, than traditional convolut…
DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
Xuxi Chen, Tianlong Chen, Weizhu Chen +3
Gigantic pre-trained models have become central to natural language processing (NLP), serving as the starting point for fine-tuning towards a range of downstream tasks. However, tw…
MetaXT: Meta Cross-Task Transfer between Disparate Label Spaces
Srinagesh Sharma, Guoqing Zheng, Ahmed Hassan Awadallah
Albeit the universal representational power of pre-trained language models, adapting them onto a specific NLP task still requires a considerably large amount of labeled data. Effec…
Fairness via Representation Neutralization
Mengnan Du, Subhabrata Mukherjee, Guanchu Wang +3
Existing bias mitigation methods for DNN models primarily work on learning debiased encoders. This process not only requires a lot of instance-level annotations for sensitive attri…