4 papers · 1 filter
Continual Memorization of Factoids in Language Models
Howard Chen, Jiayi Geng, Adithya Bhaskar +2
As new knowledge rapidly accumulates, language models (LMs) with pretrained knowledge quickly become obsolete. A common approach to updating LMs is fine-tuning them directly on new…
Extracting Rule-based Descriptions of Attention Features in Transformers
Dan Friedman, Adithya Bhaskar, Alexander Wettig +1
Mechanistic interpretability strives to explain model behavior in terms of bottom-up primitives. The leading paradigm is to express hidden states as a sparse linear combination of…
Language Models that Think, Chat Better
Adithya Bhaskar, Xi Ye, Danqi Chen
Reinforcement learning with verifiable rewards (RLVR) trains language models to use long chain-of-thought reasoning (CoT) in domains like mathematics and code with rule-based verif…
Representing Rule-based Chatbots with Transformers
Dan Friedman, Abhishek Panigrahi, Danqi Chen
What kind of internal mechanisms might Transformers use to conduct fluid, natural-sounding conversations? Prior work has illustrated by construction how Transformers can solve vari…