2.1k citations · 6.1k across the 19 of their papers we have counts for
19 papers
Brainformers: Trading Simplicity for Efficiency
Yanqi Zhou, Nan Du, Yanping Huang +12
Transformers are central to recent successes in natural language processing and computer vision. Transformers have a mostly uniform backbone where layers alternate between feed-for…
Efficiently Scaling Transformer Inference
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery +7
We study the problem of efficient generative inference for Transformer models, in one of its most challenging settings: large deep models, with tight latency targets and long seque…
Scaling Instruction-Finetuned Language Models
Hyung Won Chung, Le Hou, Shayne Longpre +32
Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we expl…
A Review of Sparse Expert Models in Deep Learning
William Fedus, Jeff Dean, Barret Zoph
Sparse expert models are a thirty-year old concept re-emerging as a popular architecture in deep learning. This class of architecture encompasses Mixture-of-Experts, Switch Transfo…
Emergent Abilities of Large Language Models
Jason Wei, Yi Tay, Rishi Bommasani +13
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictabl…
muNet: Evolving Pretrained Deep Neural Networks into Scalable Auto-tuning Multitask Systems
Andrea Gesmundo, Jeff Dean
Most uses of machine learning today involve training a model from scratch for a particular task, or sometimes starting with a model pretrained on a related task and then fine-tunin…