51 citations · 123 across the 15 of their papers we have counts for
4 papers · 1 filter
Constructing Efficient Fact-Storing MLPs for Transformers
Owen Dugan, Roberto Garcia, Ronny Junkins +5
The success of large language models (LLMs) can be attributed in part to their ability to efficiently store factual knowledge as key-value mappings within their MLP parameters. Rec…
Cartridges: Lightweight and general-purpose long context representations via self-study
Sabri Eyuboglu, Ryan Ehrlich, Simran Arora +8
Large language models are often used to answer queries grounded in large text corpora (e.g. codebases, legal documents, or chat histories) by placing the entire corpus in the conte…
Adaptive Rank Allocation: Speeding Up Modern Transformers with RaNA Adapters
Roberto Garcia, Jerry Liu, Daniel Sorvisto +1
Large Language Models (LLMs) are computationally intensive, particularly during inference. Neuron-adaptive techniques, which selectively activate neurons in Multi-Layer Perceptron…
Minions: Cost-efficient Collaboration Between On-device and Cloud Language Models
Avanika Narayan, Dan Biderman, Sabri Eyuboglu +4
We investigate an emerging setup in which a small, on-device language model (LM) with access to local data communicates with a frontier, cloud-hosted LM to solve real-world tasks i…