2.1k citations · 5.7k across the 22 of their papers we have counts for
Showing 2021Show all
2 papers · 1 filter
cs.LG2021★ 3 cited
Sparse is Enough in Scaling Transformers
Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin +4
Large Transformer models yield impressive results on many tasks, but are expensive to train, or even fine-tune, and so slow at decoding that their use and study becomes out of reac…
cs.LG2021★ 16 cited
DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning
Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery +5
The Mixture-of-Experts (MoE) architecture is showing promising results in improving parameter sharing in multi-task learning (MTL) and in scaling high-capacity neural networks. Sta…