1 citations · 1 across the 10 of their papers we have counts for
10 papers
MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers
Roberto Garcia, Jerry Liu, Ronny Junkins +3
Large language models (LLMs) store factual knowledge in their parameters. While recent work has shown that this knowledge resides in MLP layers, existing constructive and mechanist…
Olmix: A Framework for Data Mixing Throughout LM Development
Mayee F. Chen, Tyler Murray, David Heineman +5
Data mixing -- determining the ratios of data from different domains -- is a first-order concern for training language models (LMs). While existing mixing methods show promise, the…
An Information Theoretic Perspective on Agentic System Design
Shizhe He, Avanika Narayan, Ishan S. Khare +3
Agentic language model (LM) systems power modern applications like "Deep Research" and "Claude Code," and leverage multi-LM architectures to overcome context limitations. Beneath t…
Constructing Efficient Fact-Storing MLPs for Transformers
Owen Dugan, Roberto Garcia, Ronny Junkins +5
The success of large language models (LLMs) can be attributed in part to their ability to efficiently store factual knowledge as key-value mappings within their MLP parameters. Rec…
Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin +12
Large language model (LLM) queries are predominantly processed by frontier models in centralized cloud infrastructure. Demand growth strains this paradigm faster than providers can…
A Unifying Framework for Parallelizing Sequential Models with Linear Dynamical Systems
Xavier Gonzalez, E. Kelly Buchanan, Hyun Dong Lee +6
Harnessing parallelism in seemingly sequential models is a central challenge for modern machine learning. Several approaches have been proposed for evaluating sequential processes…
Smoothie: Label Free Language Model Routing
Neel Guha, Mayee F. Chen, Trevor Chow +2
Large language models (LLMs) are increasingly used in applications where LLM inputs may span many different tasks. Recent work has found that the choice of LLM is consequential, an…
Aioli: A Unified Optimization Framework for Language Model Data Mixing
Mayee F. Chen, Michael Y. Hu, Nicholas Lourie +2
Language model performance depends on identifying the optimal mixture of data groups to train on (e.g., law, code, math). Prior work has proposed a diverse set of methods to effici…
Automated Rewards via LLM-Generated Progress Functions
Vishnu Sarukkai, Brennan Shacklett, Zander Majercik +3
Large Language Models (LLMs) have the potential to automate reward engineering by leveraging their broad domain knowledge across various tasks. However, they often need many iterat…
Cookbook: A framework for improving LLM generative abilities via programmatic data generating templates
Avanika Narayan, Mayee F. Chen, Kush Bhatia +1
Fine-tuning large language models (LLMs) on instruction datasets is a common way to improve their generative capabilities. However, instruction datasets can be expensive and time-c…