1 citations · 1 across the 8 of their papers we have counts for
Showing 2024 · cs.LGShow all
3 papers · 2 filters
cs.LG2024
MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts
Rachel S. Y. Teo, Tan M. Nguyen
Sparse Mixture of Experts (SMoE) has become the key to unlocking unparalleled scalability in deep learning. SMoE has the potential to exponentially increase parameter count while m…
cs.LG2024
Elliptical Attention
Stefan K. Nielsen, Laziz U. Abdullaev, Rachel S. Y. Teo +1
Pairwise dot-product self-attention is key to the success of transformers that achieve state-of-the-art performance across a variety of applications in language and vision. This do…
cs.LG2024★ 1 cited
Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis
Rachel S. Y. Teo, Tan M. Nguyen
The remarkable success of transformers in sequence modeling tasks, spanning various applications in natural language processing and computer vision, is attributed to the critical r…