Geometric Factual Recall in Transformers

Shauli Ravfogel, Gilad Yehudai, Joan Bruna +1

How do transformer language models memorize factual associations? A common view casts internal weight matrices as associative memories over pairs of embeddings, requiring parameter…