2.3k citations · 4.2k across the 44 of their papers we have counts for
46 papers
Focused Transformer: Contrastive Training for Context Scaling
Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek +3
Large language models have an exceptional capability to incorporate new information in a contextual manner. However, the full potential of such an approach is often restrained due…
Length Generalization in Arithmetic Transformers
Samy Jelassi, Stéphane d'Ascoli, Carles Domingo-Enrich +3
We examine how transformers cope with two challenges: learning basic integer arithmetic, and generalizing to longer sequences than seen during training. We find that relative posit…
Evaluating Language Models for Mathematics through Interactions
Katherine M. Collins, Albert Q. Jiang, Simon Frieder +11
There is much excitement about the opportunity to harness the power of large language models (LLMs) when building problem-solving assistants. However, the standard methodology of e…
Lexinvariant Language Models
Qian Huang, Eric Zelikman, Sarah Li Chen +3
Token embeddings, a mapping from discrete lexical symbols to continuous vectors, are at the heart of any language model (LM). However, lexical symbol meanings can also be determine…
PaLM 2 Technical Report
Rohan Anil, Andrew M. Dai, Orhan Firat +125
We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 i…
Magnushammer: A Transformer-Based Approach to Premise Selection
Maciej Mikuła, Szymon Tworkowski, Szymon Antoniak +7
This paper presents a novel approach to premise selection, a crucial reasoning task in automated theorem proving. Traditionally, symbolic methods that rely on extensive domain know…
Path Independent Equilibrium Models Can Better Exploit Test-Time Computation
Cem Anil, Ashwini Pokle, Kaiqu Liang +5
Designing networks capable of attaining better performance with an increased inference budget is important to facilitate generalization to harder problem instances. Recent efforts…
Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs
Albert Q. Jiang, Sean Welleck, Jin Peng Zhou +6
The formalization of existing mathematical proofs is a notoriously difficult process. Despite decades of research on automation and proof assistants, writing formal proofs remains…
Leveraging Large (Visual) Language Models for Robot 3D Scene Understanding
William Chen, Siyi Hu, Rajat Talak +1
Abstract semantic 3D scene understanding is a problem of critical importance in robotics. As robots still lack the common-sense knowledge about household objects and locations of a…
Language Model Cascades
David Dohan, Winnie Xu, Aitor Lewkowycz +9
Prompted models have demonstrated impressive few-shot learning abilities. Repeated interactions at test-time with a single model, or the composition of multiple models together, fu…
Exploring Length Generalization in Large Language Models
Cem Anil, Yuhuai Wu, Anders Andreassen +7
The ability to extrapolate from short problem instances to longer ones is an important form of out-of-distribution generalization in reasoning tasks, and is crucial when learning f…
Solving Quantitative Reasoning Problems with Language Models
Aitor Lewkowycz, Anders Andreassen, David Dohan +11
Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally stru…
Insights into Pre-training via Simpler Synthetic Tasks
Yuhuai Wu, Felix Li, Percy Liang
Pre-training produces representations that are effective for a wide range of downstream tasks, but it is still unclear what properties of pre-training are necessary for effective g…
Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search
Michał Zawalski, Michał Tyrolski, Konrad Czechowski +6
Complex reasoning problems contain states that vary in the computational cost required to determine a good action plan. Taking advantage of this property, we propose Adaptive Subgo…
Autoformalization with Large Language Models
Yuhuai Wu, Albert Q. Jiang, Wenda Li +4
Autoformalization is the process of automatically translating from natural language mathematics to formal specifications and proofs. A successful autoformalization system could adv…
Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Provers
Albert Q. Jiang, Wenda Li, Szymon Tworkowski +5
In theorem proving, the task of selecting useful premises from a large library to unlock the proof of a given conjecture is crucially important. This presents a challenge for all t…
Memorizing Transformers
Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins +1
Language models typically need to be trained or finetuned in order to acquire new knowledge, which involves updating their weights. We instead envision language models that can sim…
STaR: Bootstrapping Reasoning With Reasoning
Eric Zelikman, Yuhuai Wu, Jesse Mu +1
Generating step-by-step "chain-of-thought" rationales improves language model performance on complex reasoning tasks like mathematics or commonsense question-answering. However, in…
Block-Recurrent Transformers
DeLesley Hutchins, Imanol Schlag, Yuhuai Wu +2
We introduce the Block-Recurrent Transformer, which applies a transformer layer in a recurrent fashion along a sequence, and has linear complexity with respect to sequence length.…
Hierarchical Transformers Are More Efficient Language Models
Piotr Nawrot, Szymon Tworkowski, Michał Tyrolski +4
Transformer models yield impressive results on many NLP and sequence modeling tasks. Remarkably, Transformers can handle long sequences which allows them to produce long coherent o…
Learning to Give Checkable Answers with Prover-Verifier Games
Cem Anil, Guodong Zhang, Yuhuai Wu +1
Our ability to know when to trust the decisions made by machine learning systems has not kept up with the staggering improvements in their performance, limiting their applicability…
Subgoal Search For Complex Reasoning Tasks
Konrad Czechowski, Tomasz Odrzygóźdź, Marek Zbysiński +5
Humans excel in solving complex reasoning tasks through a mental process of moving from one idea to a related one. Inspired by this, we propose Subgoal Search (kSubS) method. Its k…
On the Opportunities and Risks of Foundation Models
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli +111
AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.…
Nonlinear Invariant Risk Minimization: A Causal Approach
Chaochao Lu, Yuhuai Wu, Jośe Miguel Hernández-Lobato +1
Due to spurious correlations, machine learning systems often fail to generalize to environments whose distributions differ from the ones used at training time. Prior work addressin…