1 citations · 1 across the 8 of their papers we have counts for
7 papers · 1 filter
Almost Asymptotically Optimal Active Clustering Through Pairwise Observations
Rachel S. Y. Teo, P. N. Karthik, Ramya Korlakai Vinayak +1
We propose a new analysis framework for clustering items into an unknown number of distinct groups using noisy and actively collected responses. At each time step, an agent…
Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures
Minh-Khoi Nguyen-Nhat, Rachel S. Y. Teo, Laziz Abdullaev +3
Sparse Mixture of Experts (SMoE) has emerged as a promising solution to achieving unparalleled scalability in deep learning by decoupling model parameter count from computational c…
CAMEx: Curvature-aware Merging of Experts
Dung V. Nguyen, Minh H. Nguyen, Luc Q. Nguyen +3
Existing methods for merging experts during model training and fine-tuning predominantly rely on Euclidean geometry, which assumes a flat parameter space. This assumption can limit…
Tight Clusters Make Specialized Experts
Stefan K. Nielsen, Rachel S. Y. Teo, Laziz U. Abdullaev +1
Sparse Mixture-of-Experts (MoE) architectures have emerged as a promising approach to decoupling model capacity from computational cost. At the core of the MoE model is the router,…
MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts
Rachel S. Y. Teo, Tan M. Nguyen
Sparse Mixture of Experts (SMoE) has become the key to unlocking unparalleled scalability in deep learning. SMoE has the potential to exponentially increase parameter count while m…
Elliptical Attention
Stefan K. Nielsen, Laziz U. Abdullaev, Rachel S. Y. Teo +1
Pairwise dot-product self-attention is key to the success of transformers that achieve state-of-the-art performance across a variety of applications in language and vision. This do…
Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis
Rachel S. Y. Teo, Tan M. Nguyen
The remarkable success of transformers in sequence modeling tasks, spanning various applications in natural language processing and computer vision, is attributed to the critical r…