4 citations · 23 across the 39 of their papers we have counts for
17 papers · 1 filter
Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function
Tuan Minh Pham, Thinh Cao, Viet Nguyen +3
The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks…
On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts
Fanqi Yan, Huy Nguyen, Dung Le +3
The softmax-contaminated mixture of experts (MoE) model is deployed when a large-scale pre-trained model, which plays the role of a fixed expert, is fine-tuned for learning downstr…
Convergence Rates for Softmax Gating Mixture of Experts
Huy Nguyen, Nhat Ho, Alessandro Rinaldo
Mixture of experts (MoE) has recently emerged as an effective framework to advance the efficiency and scalability of machine learning models by softly dividing complex tasks among…
Quadratic Gating Mixture of Experts: Statistical Insights into Self-Attention
Pedram Akbarian, Huy Nguyen, Xing Han +1
Mixture of Experts (MoE) models are well known for effectively scaling model capacity while preserving computational overheads. In this paper, we establish a rigorous relation betw…
On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions
Huy Nguyen, Xing Han, Carl Harris +2
With the growing prominence of the Mixture of Experts (MoE) architecture in developing large-scale foundation models, we investigate the Hierarchical Mixture of Experts (HMoE), a s…
Statistical Advantages of Perturbing Cosine Router in Mixture of Experts
Huy Nguyen, Pedram Akbarian, Trang Pham +3
The cosine router in Mixture of Experts (MoE) has recently emerged as an attractive alternative to the conventional linear router. Indeed, the cosine router demonstrates favorable…
Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts
Huy Nguyen, Nhat Ho, Alessandro Rinaldo
The softmax gating function is arguably the most popular choice in mixture of experts modeling. Despite its widespread use in practice, the softmax gating may lead to unnecessary c…
On Parameter Estimation in Deviated Gaussian Mixture of Experts
Huy Nguyen, Khai Nguyen, Nhat Ho
We consider the parameter estimation problem in the deviated Gaussian mixture of experts in which the data are generated from $(1 - λ^{\ast}) g_0(Y| X)+ λ^{\ast} \sum_{i = 1}^{k_{\…
On Least Square Estimation in Softmax Gating Mixture of Experts
Huy Nguyen, Nhat Ho, Alessandro Rinaldo
Mixture of experts (MoE) model is a statistical machine learning design that aggregates multiple expert networks using a softmax gating function in order to form a more intricate a…
Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?
Huy Nguyen, Pedram Akbarian, Nhat Ho
Dense-to-sparse gating mixture of experts (MoE) has recently become an effective alternative to a well-known sparse MoE. Rather than fixing the number of activated experts as in th…
A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts
Huy Nguyen, Pedram Akbarian, TrungTin Nguyen +1
Mixture-of-experts (MoE) model incorporates the power of multiple submodels via gating functions to achieve greater performance in numerous regression and classification applicatio…
Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts
Huy Nguyen, Pedram Akbarian, Fanqi Yan +1
Top-K sparse softmax gating mixture of experts has been widely used for scaling up massive deep-learning architectures without increasing the computational cost. Despite its popula…
Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts
Huy Nguyen, TrungTin Nguyen, Khai Nguyen +1
Originally introduced as a neural network for ensemble learning, mixture of experts (MoE) has recently become a fundamental building block of highly successful modern deep neural n…
Demystifying Softmax Gating Function in Gaussian Mixture of Experts
Huy Nguyen, TrungTin Nguyen, Nhat Ho
Understanding the parameter estimation of softmax gating Gaussian mixture of experts has remained a long-standing open problem in the literature. It is mainly due to three fundamen…
Fast Approximation of the Generalized Sliced-Wasserstein Distance
Dung Le, Huy Nguyen, Khai Nguyen +2
Generalized sliced Wasserstein distance is a variant of sliced Wasserstein distance that exploits the power of non-linear projection through a given defining function to better cap…
Hierarchical Sliced Wasserstein Distance
Khai Nguyen, Tongzheng Ren, Huy Nguyen +3
Sliced Wasserstein (SW) distance has been widely used in different application scenarios since it can be scaled to a large number of supports without suffering from the curse of di…
On Multimarginal Partial Optimal Transport: Equivalent Forms and Computational Complexity
Khang Le, Huy Nguyen, Tung Pham +1
We study the multi-marginal partial optimal transport (POT) problem between discrete (unbalanced) measures with at most supports. We first prove that we can obtain two equi…