activity
20232026
most citedDistributed Inference and Fine-tuning of Large Language Models Over The Internet

14 citations · 29 across the 7 of their papers we have counts for

collaborators
Showing 2024Show all

2 papers · 1 filter

cs.CL2024★ 7 cited

OLMoE: Open Mixture-of-Experts Language Models

Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld +21

We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input to…

cs.CL2024

Scaling Retrieval-Based Language Models with a Trillion-Token Datastore

Rulin Shao, Jacqueline He, Akari Asai +5

Scaling laws with respect to the amount of training data and the number of parameters allow us to predict the cost-benefit trade-offs of pretraining language models (LMs) in differ…