Scaling Retrieval-Based Language Models with a Trillion-Token Datastore

Rulin Shao, Jacqueline He, Akari Asai +5

Scaling laws with respect to the amount of training data and the number of parameters allow us to predict the cost-benefit trade-offs of pretraining language models (LMs) in differ…