Characterizing Mechanisms for Factual Recall in Language Models

Qinan Yu, Jack Merullo, Ellie Pavlick

Language Models (LMs) often must integrate facts they memorized in pretraining with new information that appears in a given context. These two sources can disagree, causing competi…