9.7k citations · 20.7k across the 32 of their papers we have counts for
34 papers
Sparse is Enough in Scaling Transformers
Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin +4
Large Transformer models yield impressive results on many tasks, but are expensive to train, or even fine-tune, and so slow at decoding that their use and study becomes out of reac…
Training Verifiers to Solve Math Word Problems
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian +9
State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures…
Hierarchical Transformers Are More Efficient Language Models
Piotr Nawrot, Szymon Tworkowski, Michał Tyrolski +4
Transformer models yield impressive results on many NLP and sequence modeling tasks. Remarkably, Transformers can handle long sequences which allows them to produce long coherent o…
Evaluating Large Language Models Trained on Code
Mark Chen, Jerry Tworek, Heewoo Jun +55
We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex p…
Q-Value Weighted Regression: Reinforcement Learning with Limited Data
Piotr Kozakowski, Łukasz Kaiser, Henryk Michalewski +2
Sample efficiency and performance in the offline setting have emerged as significant challenges of deep reinforcement learning. We introduce Q-Value Weighted Regression (QWR), a si…
Rethinking Attention with Performers
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan +10
We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed t…