4 citations · 10 across the 29 of their papers we have counts for
16 papers · 1 filter
ExpertMuon-Compass: Alignment-Guided Step Sizes for Mixture-of-Experts Training
Omatharv Bharat Vaidya, Ashwin Vinod, Pedram Akbarian +3
Mixture-of-experts (MoE) language models send each token to a few experts, so each expert is trained on a different part of the data, and this part changes during training. With a…
Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts
Tho Tran Huu, Huu-Tuan Nguyen, Thien-Hai Nguyen +4
Sparse Mixture-of-Experts (SMoE) architectures are now widely deployed in state-of-the-art language and vision models, where conditional routing allows scaling to very large networ…
A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts
Viet Nguyen, Tuan Minh Pham, Thinh Cao +4
Self-attention has greatly contributed to the success of the widely used Transformer architecture by enabling learning from data with long-range dependencies. In an effort to impro…
DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks
Nghiem T. Diep, Hien Dang, Tuan Truong +3
Parameter-efficient fine-tuning (PEFT) methods have become the standard paradigm for adapting large-scale models. Among these techniques, Weight-Decomposed Low-Rank Adaptation (DoR…
Hypernetwork-Driven Low-Rank Adaptation Across Attention Heads
Nghiem T. Diep, Dung Le, Tuan Truong +3
Parameter-efficient fine-tuning (PEFT) has emerged as a powerful paradigm for adapting large-scale pre-trained models to downstream tasks with minimal additional parameters. Among…
On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating
Huy Nguyen, Thong T. Doan, Quang Pham +3
Mixture of experts (MoE) methods are a key component in most large language model architectures, including the recent series of DeepSeek models. Compared to other MoE implementatio…
RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts
Tuan Truong, Chau Nguyen, Huy Nguyen +3
Low-rank Adaptation (LoRA) has emerged as a powerful method for fine-tuning large-scale foundation models. Despite its popularity, the theoretical understanding of LoRA has remaine…
On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Nghiem T. Diep, Huy Nguyen, Chau Nguyen +5
The LLaMA-Adapter has recently emerged as an efficient fine-tuning technique for LLaMA models, leveraging zero-initialized attention to stabilize training and enhance performance.…
Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective
Fanqi Yan, Huy Nguyen, Pedram Akbarian +2
At the core of the popular Transformer architecture is the self-attention mechanism, which dynamically assigns softmax weights to each input token so that the model can focus on th…
Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts
Minh Le, Anh Nguyen, Huy Nguyen +3
Visual Prompt Tuning (VPT) has proven effective for parameter-efficient adaptation of pre-trained vision models to downstream tasks by inserting task-specific learnable prompt toke…
On Barycenter Computation: Semi-Unbalanced Optimal Transport-based Method on Gaussians
Ngoc-Hai Nguyen, Dung Le, Hoang-Phi Nguyen +2
We explore a robust version of the barycenter problem among centered Gaussian probability measures, termed Semi-Unbalanced Optimal Transport (SUOT)-based Barycenter, wherein th…
Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Experts
Fanqi Yan, Huy Nguyen, Dung Le +2
We conduct the convergence analysis of parameter estimation in the contaminated mixture of experts. This model is motivated from the prompt learning problem where ones utilize prom…
Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts
Minh Le, Chau Nguyen, Huy Nguyen +3
Prompt-based techniques, such as prompt-tuning and prefix-tuning, have gained prominence for their efficiency in fine-tuning large pre-trained models. Despite their widespread adop…
Mixture of Experts Meets Prompt-Based Continual Learning
Minh Le, An Nguyen, Huy Nguyen +4
Exploiting the power of pre-trained models, prompt-based approaches stand out compared to other continual learning solutions in effectively preventing catastrophic forgetting, even…
CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition
Quang Pham, Giang Do, Huy Nguyen +8
Sparse mixture of experts (SMoE) offers an appealing solution to scale up the model complexity beyond the mean of increasing the network's depth or width. However, effective traini…
FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion
Xing Han, Huy Nguyen, Carl Harris +2
As machine learning models in critical fields increasingly grapple with multimodal data, they face the dual challenges of handling a wide array of modalities, often incomplete due…