67 citations · 86 across the 12 of their papers we have counts for
17 papers
ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research
Sohyeon Kim, Yoonho Lee, Bo Liu +11
What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-gr…
PR-Smoother: Simulator-Preserving Non-Gaussian Smoothing for Data Assimilation
Yuta Tarumi
Many physical data assimilation (DA) workflows require smoothing methods that represent non-Gaussian posteriors over physical state variables, scale to high-dimensional simulators,…
Crypto Accounting Bench: Evaluating Frontier and Open-Weight Models on Crypto-Asset Accounting Tasks
Kareem Khattab, Omar Khattab, Mohamed Ibrahem
We introduce Crypto Accounting Bench (CAB), a benchmark for assessing whether frontier and open-weight language models can reconstruct the complete journal entry that an organizati…
Carnot: Interpretable, Interactive, and Optimized Execution of Deep Research Queries
Matthew Russo, Yash Agarwal, Tianyu Li +5
Enterprises increasingly seek to query data lakes using natural language via AI-driven tools like semantic operators or deep research agents. However, the latter operates as an opa…
OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries
Diane Tchuindjo, Devavrat Shah, Omar Khattab
Retrieval benchmarks are increasingly saturating, but we argue that efficient search is far from a solved problem. We identify a class of queries we call oblique, which seek docume…
Vector Policy Optimization: Training for Diversity Improves Test-Time Search
Ryan Bahlous-Boldi, Isha Puri, Idan Shenfeld +6
Language models must now generalize out of the box to novel environments and work inside inference-scaling search procedures, such as AlphaEvolve, that select rollouts with a varie…
PEEK: Context Map as an Orientation Cache for Long-Context LLM Agents
Zhuohan Gu, Qizheng Zhang, Omar Khattab +1
Large language model (LLM) agents increasingly operate over long and recurring external contexts, like document corpora and code repositories. Across invocations, existing approach…
Meta-Harness: End-to-End Optimization of Model Harnesses
Yoonho Lee, Roshen Nair, Qizheng Zhang +3
The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, an…
Recursive Language Models
Alex L. Zhang, Tim Kraska, Omar Khattab
We study allowing large language models (LLMs) to process arbitrarily long prompts through the lens of inference-time scaling. We propose Recursive Language Models (RLMs), a genera…
Reasoning-Intensive Regression
Diane Tchuindjo, Omar Khattab
AI researchers and practitioners increasingly apply large language models (LLMs) to what we call reasoning-intensive regression (RiR), i.e., deducing subtle numerical scores from t…
FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
Nandan Thakur, Jimmy Lin, Sam Havens +3
We introduce FreshStack, a holistic framework for automatically building information retrieval (IR) evaluation benchmarks by incorporating challenging questions and answers. FreshS…
Problem-Oriented Segmentation and Retrieval: Case Study on Tutoring Conversations
Rose E. Wang, Pawan Wirawarn, Kenny Lam +2
Many open-ended conversations (e.g., tutoring lessons or business meetings) revolve around pre-defined reference materials, like worksheets or meeting bullets. To provide a framewo…
Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrieval
Sheryl Hsu, Omar Khattab, Chelsea Finn +1
The hallucinations of large language models (LLMs) are increasingly mitigated by allowing LLMs to search for information and to ground their answers in real sources. Unfortunately,…
PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensembles
Li Siyan, Vethavikashini Chithrra Raghuram, Omar Khattab +2
Users can divulge sensitive information to proprietary LLM providers, raising significant privacy concerns. While open-source models, hosted locally on the user's machine, alleviat…
Backtracing: Retrieving the Cause of the Query
Rose E. Wang, Pawan Wirawarn, Omar Khattab +2
Many online content portals allow users to ask questions to supplement their understanding (e.g., of lectures). While information retrieval (IR) systems may provide answers for suc…
Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models
Yijia Shao, Yucheng Jiang, Theodore A. Kanell +3
We study how to apply large language models to write grounded and organized long-form articles from scratch, with comparable breadth and depth to Wikipedia pages. This underexplore…
Image and Data Mining in Reticular Chemistry Using GPT-4V
Zhiling Zheng, Zhiguo He, Omar Khattab +5
The integration of artificial intelligence into scientific research has reached a new pinnacle with GPT-4V, a large language model featuring enhanced vision capabilities, accessibl…