1 citations · 1 across the 11 of their papers we have counts for
8 papers · 1 filter
Sparsity Saves Compute, Not Samples: Oracle Inequalities for Dense and Sparse Mixture of Experts
Sagar Ghosh, Huy Nguyen, Alessandro Rinaldo
Mixture-of-Experts (MoE) models combine specialized experts through an input-dependent routing mechanism, and their sparse variants underpin many modern large-scale systems. Yet th…
Characterizing Heterogeneous Rates in Finite Mixture Estimation via Partial Optimal Transport
Dung Le, Huy Nguyen, Trang Pham +2
Parameter estimation in finite mixture models can exhibit highly heterogeneous convergence behavior: locally isolated components may be estimated substantially faster than groups o…
Partial Differential Equation Barriers to Identifiability in Infinite Mixture Models
Dung Le, Nicola Bariletto, Alessandro Rinaldo +1
We study identifiability of mixing measures in infinite mixture models. We show that, in many common cases, lack of identifiability can be characterized in terms of certain differe…
Convergence Rates for Latent Mixing Measures in Infinite Homoscedastic Location-Scale Mixture Models
Nicola Bariletto, Dung Le, Alessandro Rinaldo +1
We study posterior contraction rates for mixing measures in homoscedastic location-scale mixture models with infinitely many components. While posterior convergence at the level of…
On the Geometry of Separation in Finite Gaussian Mixtures
Huy Nguyen, Dung Le, Alessandro Rinaldo +1
We study an open problem of understanding the effects of the minimum component separation on the convergence rates of parameter estimation in finite Gaussian mixtures. We address t…
On Bayesian Softmax-Gated Mixture-of-Experts Models
Nicola Bariletto, Huy Nguyen, Nhat Ho +1
Mixture-of-experts models provide a flexible framework for learning complex probabilistic input-output relationships by combining multiple expert models through an input-dependent…
A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts
Viet Nguyen, Tuan Minh Pham, Thinh Cao +4
Self-attention has greatly contributed to the success of the widely used Transformer architecture by enabling learning from data with long-range dependencies. In an effort to impro…
Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function
Tuan Minh Pham, Thinh Cao, Viet Nguyen +3
The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks…