17 citations · 28 across the 6 of their papers we have counts for
7 papers
OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning
Krista Opsahl-Ong, Arnav Singhvi, Jasmine Collins +10
We introduce OfficeQA Pro, a benchmark for evaluating AI agents on grounded, multi-document reasoning over a large and heterogeneous document corpus. The corpus consists of U.S. Tr…
KARL: Knowledge Agents via Reinforcement Learning
Jonathan D. Chang, Andrew Drozdov, Shubham Toshniwal +23
We present a system for training enterprise search agents via reinforcement learning that achieves state-of-the-art performance across a diverse suite of hard-to-verify agentic sea…
FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
Nandan Thakur, Jimmy Lin, Sam Havens +3
We introduce FreshStack, a holistic framework for automatically building information retrieval (IR) evaluation benchmarks by incorporating challenging questions and answers. FreshS…
Long Context RAG Performance of Large Language Models
Quinn Leng, Jacob Portes, Sam Havens +2
Retrieval Augmented Generation (RAG) has emerged as a crucial technique for enhancing the accuracy of Large Language Models (LLMs) by incorporating external information. With the a…
LoRA Learns Less and Forgets Less
Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz +9
Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected w…
MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining
Jacob Portes, Alex Trott, Sam Havens +6
Although BERT-style encoder models are heavily used in NLP research, many researchers do not pretrain their own BERTs from scratch due to the high cost of training. In the past hal…
LIMIT: Less Is More for Instruction Tuning Across Evaluation Paradigms
Aditi Jha, Sam Havens, Jeremy Dohmann +2
Large Language Models are traditionally finetuned on large instruction datasets. However recent studies suggest that small, high-quality datasets can suffice for general purpose in…