17 citations · 28 across the 7 of their papers we have counts for
4 papers · 1 filter
Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs
Jacob Portes, Connor Jennings, Erica Ji Yuen +2
How does retrieval performance scale with pretraining FLOPs? We benchmark retrieval performance across LLM model sizes from 125 million parameters to 7 billion parameters pretraine…
Long Context RAG Performance of Large Language Models
Quinn Leng, Jacob Portes, Sam Havens +2
Retrieval Augmented Generation (RAG) has emerged as a crucial technique for enhancing the accuracy of Large Language Models (LLMs) by incorporating external information. With the a…
LoRA Learns Less and Forgets Less
Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz +9
Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected w…
LIMIT: Less Is More for Instruction Tuning Across Evaluation Paradigms
Aditi Jha, Sam Havens, Jeremy Dohmann +2
Large Language Models are traditionally finetuned on large instruction datasets. However recent studies suggest that small, high-quality datasets can suffice for general purpose in…