14 papers · 1 filter
Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts
Tho Tran Huu, Huu-Tuan Nguyen, Thien-Hai Nguyen +4
Sparse Mixture-of-Experts (SMoE) architectures are now widely deployed in state-of-the-art language and vision models, where conditional routing allows scaling to very large networ…
Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts
Minh Le, Anh Nguyen, Huy Nguyen +3
Visual Prompt Tuning (VPT) has proven effective for parameter-efficient adaptation of pre-trained vision models to downstream tasks by inserting task-specific learnable prompt toke…
DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks
Nghiem T. Diep, Hien Dang, Tuan Truong +3
Parameter-efficient fine-tuning (PEFT) methods have become the standard paradigm for adapting large-scale models. Among these techniques, Weight-Decomposed Low-Rank Adaptation (DoR…
A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts
Viet Nguyen, Tuan Minh Pham, Thinh Cao +4
Self-attention has greatly contributed to the success of the widely used Transformer architecture by enabling learning from data with long-range dependencies. In an effort to impro…
On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating
Huy Nguyen, Thong T. Doan, Quang Pham +3
Mixture of experts (MoE) methods are a key component in most large language model architectures, including the recent series of DeepSeek models. Compared to other MoE implementatio…
Hypernetwork-Driven Low-Rank Adaptation Across Attention Heads
Nghiem T. Diep, Dung Le, Tuan Truong +3
Parameter-efficient fine-tuning (PEFT) has emerged as a powerful paradigm for adapting large-scale pre-trained models to downstream tasks with minimal additional parameters. Among…