4 citations · 23 across the 39 of their papers we have counts for
43 papers
Joint Estimation and Feature Selection in Overspecified Finite Mixture of Regressions
Abbas Khalili, Pengqi Liu, Huy Nguyen +1
We investigate joint estimation and feature selection in sparse finite mixtures of regressions with an overspecified number of mixture components or mixture order. More specificall…
Characterizing Heterogeneous Rates in Finite Mixture Estimation via Partial Optimal Transport
Dung Le, Huy Nguyen, Trang Pham +2
Parameter estimation in finite mixture models can exhibit highly heterogeneous convergence behavior: locally isolated components may be estimated substantially faster than groups o…
On the Geometry of Separation in Finite Gaussian Mixtures
Huy Nguyen, Dung Le, Alessandro Rinaldo +1
We study an open problem of understanding the effects of the minimum component separation on the convergence rates of parameter estimation in finite Gaussian mixtures. We address t…
A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts
Viet Nguyen, Tuan Minh Pham, Thinh Cao +4
Self-attention has greatly contributed to the success of the widely used Transformer architecture by enabling learning from data with long-range dependencies. In an effort to impro…
Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function
Tuan Minh Pham, Thinh Cao, Viet Nguyen +3
The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks…
Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneity
Fanqi Yan, Dung Le, Trang Pham +2
Contaminated mixture of experts (MoE) is motivated by transfer learning methods where a pre-trained model, acting as a frozen expert, is integrated with an adapter model, functioni…
The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes
Redacted by arXiv
This document consolidates publicly reported technical details about Metas Llama 4 model family. It summarizes (i) released variants (Scout and Maverick) and the broader herd conte…
DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks
Nghiem T. Diep, Hien Dang, Tuan Truong +3
Parameter-efficient fine-tuning (PEFT) methods have become the standard paradigm for adapting large-scale models. Among these techniques, Weight-Decomposed Low-Rank Adaptation (DoR…
Hypernetwork-Driven Low-Rank Adaptation Across Attention Heads
Nghiem T. Diep, Dung Le, Tuan Truong +3
Parameter-efficient fine-tuning (PEFT) has emerged as a powerful paradigm for adapting large-scale pre-trained models to downstream tasks with minimal additional parameters. Among…
One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning
Minh Le, Bao-Ngoc Dao, Huy Nguyen +3
Prompt-based methods have recently gained prominence in Continual Learning (CL) due to their strong performance and memory efficiency. A prevalent strategy in this paradigm assigns…
CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via Competition
Nam V. Nguyen, Huy Nguyen, Quang Pham +3
Sparse mixture of experts (SMoE) offers an appealing solution to scale up the model complexity beyond the mean of increasing the network's depth or width. However, we argue that ef…
On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts
Fanqi Yan, Huy Nguyen, Dung Le +3
The softmax-contaminated mixture of experts (MoE) model is deployed when a large-scale pre-trained model, which plays the role of a fixed expert, is fine-tuned for learning downstr…
On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating
Huy Nguyen, Thong T. Doan, Quang Pham +3
Mixture of experts (MoE) methods are a key component in most large language model architectures, including the recent series of DeepSeek models. Compared to other MoE implementatio…
Convergence Rates for Softmax Gating Mixture of Experts
Huy Nguyen, Nhat Ho, Alessandro Rinaldo
Mixture of experts (MoE) has recently emerged as an effective framework to advance the efficiency and scalability of machine learning models by softly dividing complex tasks among…
RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts
Tuan Truong, Chau Nguyen, Huy Nguyen +3
Low-rank Adaptation (LoRA) has emerged as a powerful method for fine-tuning large-scale foundation models. Despite its popularity, the theoretical understanding of LoRA has remaine…
On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Nghiem T. Diep, Huy Nguyen, Chau Nguyen +5
The LLaMA-Adapter has recently emerged as an efficient fine-tuning technique for LLaMA models, leveraging zero-initialized attention to stabilize training and enhance performance.…
Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective
Fanqi Yan, Huy Nguyen, Pedram Akbarian +2
At the core of the popular Transformer architecture is the self-attention mechanism, which dynamically assigns softmax weights to each input token so that the model can focus on th…
Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts
Minh Le, Anh Nguyen, Huy Nguyen +3
Visual Prompt Tuning (VPT) has proven effective for parameter-efficient adaptation of pre-trained vision models to downstream tasks by inserting task-specific learnable prompt toke…
Heuristic-Free Multi-Teacher Learning
Huy Thong Nguyen, En-Hung Chu, Lenord Melvix +3
We introduce Teacher2Task, a novel framework for multi-teacher learning that eliminates the need for manual aggregation heuristics. Existing multi-teacher methods typically rely on…
Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Experts
Fanqi Yan, Huy Nguyen, Dung Le +2
We conduct the convergence analysis of parameter estimation in the contaminated mixture of experts. This model is motivated from the prompt learning problem where ones utilize prom…
Quadratic Gating Mixture of Experts: Statistical Insights into Self-Attention
Pedram Akbarian, Huy Nguyen, Xing Han +1
Mixture of Experts (MoE) models are well known for effectively scaling model capacity while preserving computational overheads. In this paper, we establish a rigorous relation betw…
On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions
Huy Nguyen, Xing Han, Carl Harris +2
With the growing prominence of the Mixture of Experts (MoE) architecture in developing large-scale foundation models, we investigate the Hierarchical Mixture of Experts (HMoE), a s…
Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts
Minh Le, Chau Nguyen, Huy Nguyen +3
Prompt-based techniques, such as prompt-tuning and prefix-tuning, have gained prominence for their efficiency in fine-tuning large pre-trained models. Despite their widespread adop…
Statistical Advantages of Perturbing Cosine Router in Mixture of Experts
Huy Nguyen, Pedram Akbarian, Trang Pham +3
The cosine router in Mixture of Experts (MoE) has recently emerged as an attractive alternative to the conventional linear router. Indeed, the cosine router demonstrates favorable…