2.1k citations · 7.4k across the 36 of their papers we have counts for
36 papers
Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study
Menglong Cui, Pengzhi Gao, Wei Liu +2
Large language models (LLMs) have shown continuously improving multilingual capabilities, and even small-scale open-source models have demonstrated rapid performance enhancement. I…
Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance
Kaitlyn Zhou, Jena D. Hwang, Xiang Ren +3
The ability to communicate uncertainty, risk, and limitation is crucial for the safety of large language models. However, current evaluations of these abilities rely on simple cali…
Naive Bayes-based Context Extension for Large Language Models
Jianlin Su, Murtadha Ahmed, Wenbo +3
Large Language Models (LLMs) have shown promising in-context learning abilities. However, conventional In-Context Learning (ICL) approaches are often impeded by length limitations…
Language Model Cascades
David Dohan, Winnie Xu, Aitor Lewkowycz +9
Prompted models have demonstrated impressive few-shot learning abilities. Repeated interactions at test-time with a single model, or the composition of multiple models together, fu…
Exploring Length Generalization in Large Language Models
Cem Anil, Yuhuai Wu, Anders Andreassen +7
The ability to extrapolate from short problem instances to longer ones is an important form of out-of-distribution generalization in reasoning tasks, and is crucial when learning f…
Solving Quantitative Reasoning Problems with Language Models
Aitor Lewkowycz, Anders Andreassen, David Dohan +11
Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally stru…
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao +448
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabil…
PaLM: Scaling Language Modeling with Pathways
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin +64
Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of…
Scaling Up Models and Data with and
Adam Roberts, Hyung Won Chung, Anselm Levskaya +40
Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves. Scaling can…
Show Your Work: Scratchpads for Intermediate Computation with Language Models
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari +9
Large pre-trained language models perform remarkably well on tasks that can be done "in one pass", such as generating realistic text or synthesizing computer programs. However, the…
How to decay your learning rate
Aitor Lewkowycz
Complex learning rate schedules have become an integral part of deep learning. We find empirically that common fine-tuned schedules decay the learning rate after the weight norm bo…
Gravitational path integral from the deformation
Alexandre Belin, Aitor Lewkowycz, Gabor Sarosi
We study a deformation of large conformal field theories, a higher dimensional generalization of the deformation. The deformed partition function satisfies a fl…
On the training dynamics of deep networks with regularization
Aitor Lewkowycz, Guy Gur-Ari
We study the role of regularization in deep learning, and uncover simple relations between the performance of the model, the coefficient, the learning rate, and the num…
The large learning rate phase of deep learning: the catapult mechanism
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer +2
The choice of initial learning rate can have a profound effect on the performance of deep networks. We present a class of neural networks with solvable training dynamics, and confi…
and EE, with implications for (A)dS subregion encodings
Aitor Lewkowycz, Junyu Liu, Eva Silverstein +1
We initiate a study of subregion dualities, entropy, and redundant encoding of bulk points in holographic theories deformed by and its generalizations. This includes bot…
Complexity and the bulk volume, a new York time story
Alexandre Belin, Aitor Lewkowycz, Gabor Sarosi
We study the boundary description of the volume of maximal Cauchy slices using the recently derived equivalence between bulk and boundary symplectic forms. The volume of constant m…
Emergent classical spacetime from microstates of an incipient black hole
Vijay Balasubramanian, David Berenstein, Aitor Lewkowycz +3
Black holes have an enormous underlying space of microstates, but universal macroscopic physics characterized by mass, charge and angular momentum as well as a causally disconnecte…
The boundary dual of the bulk symplectic form
Alexandre Belin, Aitor Lewkowycz, Gábor Sárosi
In this paper, we study the overlaps of wavefunctionals prepared by turning on sources in the Euclidean path integral. For nearby states, these overlaps give rise to a Kahler struc…
Modular Flow as a Disentangler
Yiming Chen, Xi Dong, Aitor Lewkowycz +1
In holographic duality, the entanglement entropy of a boundary region is proposed to be dual to the area of an extremal codimension-2 surface that is homologous to the boundary reg…
Thermalization, Viscosity and the Averaged Null Energy Condition
Luca V. Delacrétaz, Thomas Hartman, Sean A. Hartnoll +1
We explore the implications of the averaged null energy condition for thermal states of relativistic quantum field theories. A key property of such thermal states is the thermaliza…
The Holographic Shape of Entanglement and Einstein's Equations
Aitor Lewkowycz, Onkar Parrikar
We study shape-deformations of the entanglement entropy and the modular Hamiltonian for an arbitrary subregion and state (with a smooth dual geometry) in a holographic conformal fi…
Inside Out: Meet The Operators Inside The Horizon
Ahmed Almheiri, Tarek Anous, Aitor Lewkowycz
Based on the work of Heemskerk, Marolf, Polchinski and Sully (HMPS), we study the reconstruction of operators behind causal horizons in time dependent geometries obtained by acting…
Entropy, Extremality, Euclidean Variations, and the Equations of Motion
Xi Dong, Aitor Lewkowycz
We study the Euclidean gravitational path integral computing the Renyi entropy and analyze its behavior under small variations. We argue that, in Einstein gravity, the extremality…
Bulk locality from modular flow
Thomas Faulkner, Aitor Lewkowycz
We study the reconstruction of bulk operators in the entanglement wedge in terms of low energy operators localized in the respective boundary region. To leading order in , the d…