565 citations · 2.2k across the 23 of their papers we have counts for
23 papers
Neuron Populations Exhibit Divergent Selectivity with Scale
Amil Dravid, Yasaman Bahri, Alexei A. Efros +1
We investigate whether neuron populations within neural networks evolve predictably with scale, extending scaling laws beyond macroscopic observables such as loss. To probe this qu…
Symmetry in language statistics shapes the geometry of model representations
Dhruva Karkada, Daniel J. Korchinski, Andres Nava +2
The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth on…
Context Structure Reshapes the Representational Geometry of Language Models
Eghbal A. Hosseini, Yuxuan Li, Yasaman Bahri +2
Large Language Models (LLMs) have been shown to organize the representations of input sequences into straighter neural trajectories in their deep layers, which has been hypothesize…
On the Emergence of Linear Analogies in Word Embeddings
Daniel J. Korchinski, Dhruva Karkada, Yasaman Bahri +1
Models such as Word2Vec and GloVe construct word embeddings based on the co-occurrence probability of words and in text corpora. The resulting vectors not on…
CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning
Hao Cui, Zahra Shamsi, Gowoon Cheon +31
Scientific problem-solving involves synthesizing information while applying expert knowledge. We introduce CURIE, a scientific long-Context Understanding,Reasoning and Information…
Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models
Dhruva Karkada, James B. Simon, Yasaman Bahri +1
Self-supervised word embedding algorithms such as word2vec provide a minimal setting for studying representation learning in language modeling. We examine the quartic Taylor approx…
Quantum Many-Body Physics Calculations with Large Language Models
Haining Pan, Nayantara Mudur, Will Taranto +5
Large language models (LLMs) have demonstrated an unprecedented ability to perform complex tasks in multiple domains, including mathematical and scientific reasoning. We demonstrat…
Les Houches Lectures on Deep Learning at Large & Infinite Width
Yasaman Bahri, Boris Hanin, Antonin Brossollet +4
These lectures, presented at the 2022 Les Houches Summer School on Statistical Physics and Machine Learning, focus on the infinite-width limit and large-width regime of deep neural…
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao +448
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabil…
The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
Anders Andreassen, Yasaman Bahri, Behnam Neyshabur +1
Although machine learning models typically experience a drop in performance on out-of-distribution data, accuracies on in- versus out-of-distribution data are widely observed to fo…
Explaining Neural Scaling Laws
Yasaman Bahri, Ethan Dyer, Jared Kaplan +2
The population loss of trained deep neural networks often follows precise power-law scaling relations with either the size of the training dataset or the number of parameters in th…
Infinite attention: NNGP and NTK for deep attention networks
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein +1
There is a growing amount of literature on the relationship between wide neural networks (NNs) and Gaussian processes (GPs), identifying an equivalence between the two for a variet…
Exact posterior distributions of wide Bayesian neural networks
Jiri Hron, Yasaman Bahri, Roman Novak +2
Recent work has shown that the prior over functions induced by a deep Bayesian neural network (BNN) behaves as a Gaussian process (GP) as the width of all layers becomes large. How…
The large learning rate phase of deep learning: the catapult mechanism
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer +2
The choice of initial learning rate can have a profound effect on the performance of deep networks. We present a class of neural networks with solvable training dynamics, and confi…
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz +4
A longstanding goal in deep learning research has been to precisely characterize training and generalization. However, the often complex loss landscapes of neural networks have mad…
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
Roman Novak, Lechao Xiao, Jaehoon Lee +6
There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs). This equivalence enables, for instance, test set pred…
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein +2
In recent years, state-of-the-art methods in computer vision have utilized increasingly deep convolutional neural network architectures (CNNs), with some of the most successful mod…
Sensitivity and Generalization in Neural Networks: an Empirical Study
Roman Novak, Yasaman Bahri, Daniel A. Abolafia +2
In practice it is often found that large over-parameterized neural networks generalize better than their smaller counterparts, an observation that appears to conflict with classica…
Deep Neural Networks as Gaussian Processes
Jaehoon Lee, Yasaman Bahri, Roman Novak +3
It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinit…
Phonon analogue of topological nodal semimetals
Hoi Chun Po, Yasaman Bahri, Ashvin Vishwanath
Topological band structures in electronic systems like topological insulators and semimetals give rise to highly unusual physical properties. Analogous topological effects have als…
Stable non-Fermi liquid phase of itinerant spin-orbit coupled ferromagnets
Yasaman Bahri, Andrew C. Potter
Direct coupling between gapless bosons and a Fermi surface results in the destruction of Landau quasiparticles and a breakdown of Fermi liquid theory. Such a non-Fermi liquid phase…
Localization and topology protected quantum coherence at the edge of 'hot' matter
Yasaman Bahri, Ronen Vosk, Ehud Altman +1
Topological phases are often characterized by special edge states confined near the boundaries by an energy gap in the bulk. On raising temperature, these edge states are lost in a…
Detecting Majorana fermions in quasi-one-dimensional topological phases using nonlocal order parameters
Yasaman Bahri, Ashvin Vishwanath
Topological phases which host Majorana fermions can not be identified via local order parameters. We give simple nonlocal order parameters to distinguish quasi-one-dimensional (1D)…