Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs

Jacob Portes, Connor Jennings, Erica Ji Yuen +2

How does retrieval performance scale with pretraining FLOPs? We benchmark retrieval performance across LLM model sizes from 125 million parameters to 7 billion parameters pretraine…