5 papers
Linear representations in language models can change dramatically over a conversation
Andrew Kyle Lampinen, Yuxuan Li, Eghbal Hosseini +2
Language model representations often contain linear directions that correspond to high-level concepts. Here, we study the dynamics of these representations: how representations evo…
Context Structure Reshapes the Representational Geometry of Language Models
Eghbal A. Hosseini, Yuxuan Li, Yasaman Bahri +2
Large Language Models (LLMs) have been shown to organize the representations of input sequences into straighter neural trajectories in their deep layers, which has been hypothesize…
Latent learning: episodic memory complements parametric learning by enabling flexible reuse of experiences
Andrew Kyle Lampinen, Martin Engelcke, Yuxuan Li +2
When do machine learning systems fail to generalize, and what mechanisms could improve their generalization? Here, we draw inspiration from cognitive science to argue that one weak…
Just-in-time and distributed task representations in language models
Yuxuan Li, Declan Campbell, Stephanie C. Y. Chan +1
Many of language models' impressive capabilities originate from their in-context learning: based on instructions or examples, they can infer and perform new tasks without weight up…
Representation biases: will we achieve complete understanding by analyzing representations?
Andrew Kyle Lampinen, Stephanie C. Y. Chan, Yuxuan Li +1
A common approach in neuroscience is to study neural representations as a means to understand a system -- increasingly, by relating the neural representations to the internal repre…