1 citations · 1 across the 3 of their papers we have counts for
8 papers
Symmetry in language statistics shapes the geometry of model representations
Dhruva Karkada, Daniel J. Korchinski, Andres Nava +2
The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth on…
Neuron Populations Exhibit Divergent Selectivity with Scale
Amil Dravid, Yasaman Bahri, Alexei A. Efros +1
We investigate whether neuron populations within neural networks evolve predictably with scale, extending scaling laws beyond macroscopic observables such as loss. To probe this qu…
Context Structure Reshapes the Representational Geometry of Language Models
Eghbal A. Hosseini, Yuxuan Li, Yasaman Bahri +2
Large Language Models (LLMs) have been shown to organize the representations of input sequences into straighter neural trajectories in their deep layers, which has been hypothesize…
On the Emergence of Linear Analogies in Word Embeddings
Daniel J. Korchinski, Dhruva Karkada, Yasaman Bahri +1
Models such as Word2Vec and GloVe construct word embeddings based on the co-occurrence probability of words and in text corpora. The resulting vectors not on…
Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models
Dhruva Karkada, James B. Simon, Yasaman Bahri +1
Self-supervised word embedding algorithms such as word2vec provide a minimal setting for studying representation learning in language modeling. We examine the quartic Taylor approx…
CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning
Hao Cui, Zahra Shamsi, Gowoon Cheon +31
Scientific problem-solving involves synthesizing information while applying expert knowledge. We introduce CURIE, a scientific long-Context Understanding,Reasoning and Information…