4 citations · 6 across the 14 of their papers we have counts for
7 papers · 1 filter
Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls
Xiaoyan Bai, Itamar Pres, Yuntian Deng +5
Language models are increasingly capable, yet still fail at a seemingly simple task of multi-digit multiplication. In this work, we study why, by reverse-engineering a model that s…
Chronotome: Real-Time Topic Modeling for Streaming Embedding Spaces
Matte Lim, Catherine Yeh, Martin Wattenberg +2
Many real-world datasets -- from an artist's body of work to a person's social media history -- exhibit meaningful semantic changes over time that are difficult to capture with exi…
Story Ribbons: Reimagining Storyline Visualizations with Large Language Models
Catherine Yeh, Tara Menon, Robin Singh Arya +4
Analyzing literature involves tracking interactions between characters, locations, and themes. Visualization has the potential to facilitate the mapping and analysis of these compl…
What Does it Mean for a Neural Network to Learn a "World Model"?
Kenneth Li, Fernanda Viégas, Martin Wattenberg
We propose a set of precise criteria for saying a neural net learns and uses a "world model." The goal is to give an operational meaning to terms that are often used informally, in…
The Geometry of Self-Verification in a Task-Specific Reasoning Model
Andrew Lee, Lihao Sun, Chris Wendler +2
How do reasoning models verify their own answers? We study this question by training a model using DeepSeek R1's recipe on the CountDown task. We leverage the fact that preference…
When Bad Data Leads to Good Models
Kenneth Li, Yida Chen, Fernanda Viégas +1
In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of "quality" from the perspective of pre- an…
Shared Global and Local Geometry of Language Model Embeddings
Andrew Lee, Melanie Weber, Fernanda Viégas +1
Researchers have recently suggested that models share common representations. In our work, we find numerous geometric similarities across the token embeddings of large language mod…