8 papers
MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers
Roberto Garcia, Jerry Liu, Ronny Junkins +3
Large language models (LLMs) store factual knowledge in their parameters. While recent work has shown that this knowledge resides in MLP layers, existing constructive and mechanist…
Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin +13
Large language model (LLM) queries are predominantly processed by frontier models in centralized cloud infrastructure. Demand growth strains this paradigm faster than providers can…
A Unifying Framework for Parallelizing Sequential Models with Linear Dynamical Systems
Xavier Gonzalez, E. Kelly Buchanan, Hyun Dong Lee +6
Harnessing parallelism in seemingly sequential models is a central challenge for modern machine learning. Several approaches have been proposed for evaluating sequential processes…
Olmix: A Framework for Data Mixing Throughout LM Development
Mayee F. Chen, Tyler Murray, David Heineman +5
Data mixing -- determining the ratios of data from different domains -- is a first-order concern for training language models (LMs). While existing mixing methods show promise, the…
An Information Theoretic Perspective on Agentic System Design
Shizhe He, Avanika Narayan, Ishan S. Khare +3
Agentic language model (LM) systems power modern applications like "Deep Research" and "Claude Code," and leverage multi-LM architectures to overcome context limitations. Beneath t…
Constructing Efficient Fact-Storing MLPs for Transformers
Owen Dugan, Roberto Garcia, Ronny Junkins +5
The success of large language models (LLMs) can be attributed in part to their ability to efficiently store factual knowledge as key-value mappings within their MLP parameters. Rec…