1 citations · 1 across the 11 of their papers we have counts for
15 papers
Sparsity Saves Compute, Not Samples: Oracle Inequalities for Dense and Sparse Mixture of Experts
Sagar Ghosh, Huy Nguyen, Alessandro Rinaldo
Mixture-of-Experts (MoE) models combine specialized experts through an input-dependent routing mechanism, and their sparse variants underpin many modern large-scale systems. Yet th…
Characterizing Heterogeneous Rates in Finite Mixture Estimation via Partial Optimal Transport
Dung Le, Huy Nguyen, Trang Pham +2
Parameter estimation in finite mixture models can exhibit highly heterogeneous convergence behavior: locally isolated components may be estimated substantially faster than groups o…
Partial Differential Equation Barriers to Identifiability in Infinite Mixture Models
Dung Le, Nicola Bariletto, Alessandro Rinaldo +1
We study identifiability of mixing measures in infinite mixture models. We show that, in many common cases, lack of identifiability can be characterized in terms of certain differe…
Convergence Rates for Latent Mixing Measures in Infinite Homoscedastic Location-Scale Mixture Models
Nicola Bariletto, Dung Le, Alessandro Rinaldo +1
We study posterior contraction rates for mixing measures in homoscedastic location-scale mixture models with infinitely many components. While posterior convergence at the level of…
On the Geometry of Separation in Finite Gaussian Mixtures
Huy Nguyen, Dung Le, Alessandro Rinaldo +1
We study an open problem of understanding the effects of the minimum component separation on the convergence rates of parameter estimation in finite Gaussian mixtures. We address t…
On Bayesian Softmax-Gated Mixture-of-Experts Models
Nicola Bariletto, Huy Nguyen, Nhat Ho +1
Mixture-of-experts models provide a flexible framework for learning complex probabilistic input-output relationships by combining multiple expert models through an input-dependent…
A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts
Viet Nguyen, Tuan Minh Pham, Thinh Cao +4
Self-attention has greatly contributed to the success of the widely used Transformer architecture by enabling learning from data with long-range dependencies. In an effort to impro…
Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function
Tuan Minh Pham, Thinh Cao, Viet Nguyen +3
The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks…
Conformalized Bayesian Inference, with Applications to Random Partition Models
Nicola Bariletto, Nhat Ho, Alessandro Rinaldo
Bayesian posterior distributions naturally represent parameter uncertainty informed by data. However, when the parameter space is complex, as in many nonparametric settings where i…
On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts
Fanqi Yan, Huy Nguyen, Dung Le +3
The softmax-contaminated mixture of experts (MoE) model is deployed when a large-scale pre-trained model, which plays the role of a fixed expert, is fine-tuned for learning downstr…
On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating
Huy Nguyen, Thong T. Doan, Quang Pham +3
Mixture of experts (MoE) methods are a key component in most large language model architectures, including the recent series of DeepSeek models. Compared to other MoE implementatio…
Convergence Rates for Softmax Gating Mixture of Experts
Huy Nguyen, Nhat Ho, Alessandro Rinaldo
Mixture of experts (MoE) has recently emerged as an effective framework to advance the efficiency and scalability of machine learning models by softly dividing complex tasks among…
Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective
Fanqi Yan, Huy Nguyen, Pedram Akbarian +2
At the core of the popular Transformer architecture is the self-attention mechanism, which dynamically assigns softmax weights to each input token so that the model can focus on th…
Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts
Huy Nguyen, Nhat Ho, Alessandro Rinaldo
The softmax gating function is arguably the most popular choice in mixture of experts modeling. Despite its widespread use in practice, the softmax gating may lead to unnecessary c…
On Least Square Estimation in Softmax Gating Mixture of Experts
Huy Nguyen, Nhat Ho, Alessandro Rinaldo
Mixture of experts (MoE) model is a statistical machine learning design that aggregates multiple expert networks using a softmax gating function in order to form a more intricate a…