Showing cs.LGShow all
2 papers · 1 filter
cs.LG2025
Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective
Fanqi Yan, Huy Nguyen, Pedram Akbarian +2
At the core of the popular Transformer architecture is the self-attention mechanism, which dynamically assigns softmax weights to each input token so that the model can focus on th…
cs.LG2024
Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Experts
Fanqi Yan, Huy Nguyen, Dung Le +2
We conduct the convergence analysis of parameter estimation in the contaminated mixture of experts. This model is motivated from the prompt learning problem where ones utilize prom…