4 papers · 1 filter
MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers
Roberto Garcia, Jerry Liu, Ronny Junkins +3
Large language models (LLMs) store factual knowledge in their parameters. While recent work has shown that this knowledge resides in MLP layers, existing constructive and mechanist…
Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin +13
Large language model (LLM) queries are predominantly processed by frontier models in centralized cloud infrastructure. Demand growth strains this paradigm faster than providers can…
A Unifying Framework for Parallelizing Sequential Models with Linear Dynamical Systems
Xavier Gonzalez, E. Kelly Buchanan, Hyun Dong Lee +6
Harnessing parallelism in seemingly sequential models is a central challenge for modern machine learning. Several approaches have been proposed for evaluating sequential processes…
Olmix: A Framework for Data Mixing Throughout LM Development
Mayee F. Chen, Tyler Murray, David Heineman +5
Data mixing -- determining the ratios of data from different domains -- is a first-order concern for training language models (LMs). While existing mixing methods show promise, the…