MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers

Roberto Garcia, Jerry Liu, Ronny Junkins +3

Large language models (LLMs) store factual knowledge in their parameters. While recent work has shown that this knowledge resides in MLP layers, existing constructive and mechanist…